Math Reading Group

[sullen estuary]

A reading group for learning the Math found in ML papers (or not)

Official Github Repo: https://github.com/irregular-rhomboid/EAI-Math-Reading-Group
Playlist of past recordings: https://www.youtube.com/playlist?list=PLvtrkEledFjodK_UKua2h4exNMcWNGJrs

<@&1098008274361663629> Hey all. Since we have a newly christened math rg thread, I'm reposting the announcement for this sunday's session on gradient dualization. (see #1089231591785639946 message)

I just created an event for it: https://discord.com/events/729741769192767510/1309611044754427944

@sinful forge and @royal osprey will be talking about the recent work on gradient dualization and operator norms.

The recommended reading, in order of difficulty is

• https://arxiv.org/abs/2409.20325
• https://proceedings.neurips.cc/paper_files/paper/2015/file/f50a6c02a3fc5a3a5d4d9391f05f3efc-Paper.pdf
• https://arxiv.org/abs/2410.21265
• https://arxiv.org/abs/2310.17813
• https://arxiv.org/abs/1708.00523

Please be aware that the recording of this session will only be available for a limited amount of time, so if you are interested in the topic, the best option is to attend the session live.

[reef estuary]

What is the prerequisite for this?

[sullen estuary]

if you want the bare minimum beyond what's typically covered when learning ML, its

• the notion of dual space from linear algebra https://en.wikipedia.org/wiki/Dual_space
• Normed spaces in finite dimensions, and the notion of "topological dual" https://en.wikipedia.org/wiki/Normed_vector_space
• Matrix/Operator norms https://en.wikipedia.org/wiki/Matrix_norm

[reef estuary]

@sullen estuary

1. I remember mods say that the reading group will be recorded and then taken down, may I ask for a link of the recorded video?

2. Also one statement used to motivate the use of spectral norm for optimizers was:
   ||del Y|| <= ||del W||*||X||

Where ||del W|| is spectral norm and others are L2 norm. Can you cite this? Or give context for this?

Thank you very much

[sullen estuary]

1. I still need to edit and upload the recording. Coming soon™️
2. you can find that in one of the first papers listed in the announcement. This is basically a consequence of the definition of operator norms/matrix norms.The spectral norm is the operator norm when the norm on the input and output space are both the L2 norm

[reef estuary]

@sullen estuary Does this imply that the more alignment is there between the input and weights, the better shampoo will work?

[dusky marsh]

When you say alignment, do you mean an inner product or what?

[reef estuary]

Yes

[reef estuary]

Or rather the more alignment between input and weights there is, the more difficult time adam has but shampoo will make it work?

[unique axle]

alignment was heavily discussed in https://arxiv.org/abs/2407.05872

[dusky marsh]

Reminds me of https://en.m.wikipedia.org/wiki/Good_regulator

[tulip gazelle]

@sinful forge @royal osprey what's a good pedagogical source for the details of the norm duality and constructions with different layers we were discussing at the end of the call?

[sterile gate]

(sorry if maybe very beginner question but) isn't this basically solomonoff induction? if the simplest algorithm that can control outputs by manipulating inputs must be an accurate model of the system, the simplest algorithm that can predict outputs based on inputs must also be an accurate model of the system, no?

[dusky marsh]

It isn't accurate in general to say that a map regulates the territory. Maybe you had something else in mind. There might be a different way to describe things that makes that work.

Regulating means controlling or manipulating.

The existence of that theorem is the only thing I know about cybernetics so don't lean on my knowledge too much.

[sterile gate]

fair enough!

[tulip gazelle]

I'm currently using "Modular Duality in Deep Learning" btw

[sullen estuary]

That's a good starting point imo. There's also "old optimizer, new norm, an anthology" which is very much pedagogical

[tulip gazelle]

One of the worst things about chronic migraines is what it does to my memory. I spent five hours yesterday reinventing [the analysis I did off-the-cuff](#research message) a month ago >.>

On the bright side I now strongly believe that empirical investigation into the distribution of ||Tv||/||v|| in a trained model using semantically meaningful inputs is a good idea and have written this fact down in three locations so as to (hopefully) not forget it again.

[dusky marsh]

It might be accurate to say that a model regulates inputs by manipulating them until they are accurate outputs.

A model could be considered a map of how to turn a map of one part of the territory, the input data, into a different map of the territory, the output data.

[sullen estuary]

<@&1098008274361663629> The recording is finally here. Y'all can thank @pliant peak for having recorded on his end.

Note that it is unlisted, and will be removed after a week. https://youtu.be/ZZPHm8eASOE

cc @sinful forge @royal osprey

[gloomy cloud]

what are the extra insights of this talk, for people who have already read the papers?

[sullen estuary]

Not much I would say. There was some back and forth between franz and the jeremy about some of the questions that had been discussed previously, around the middle of the talk

[slow parcel]

Thanks for posting this!

[gloomy cloud]

my current thought about orthonormalization replacing Adam is that all of its benefits come from the rows and columns being equal-sized (like Adafactor). I feel that pairs of rows being orthogonal is a downside

a mental testcase is a 2-input, 2-output model with a 2x2 matrix. the first channel of the input is scaled by 3x, and the first channel of the output is scaled by 3x. Adam can solve this perfectly. but if you orthonormalize the gradients, there is a lot of extra noise

[sullen estuary]

Hey all, I'm thinking of restarting the reading group after new years. The goal would be to get into optimization (classical and stochastic), but before I want to do some category theory as a (mostly) shitpost. The question is whether y'all would prefer to get the basics from a textbook (I have found a good one already) or directly read some of the Cats4AI papers

<@&1098008274361663629>

[sullen estuary]

Well, the people have spoken. I'll prepare some papers to read in the coming days

[sullen estuary]

Alright <@&1098008274361663629> Next meeting on <t:1736715600>. We will be talking about some Category Theory.

The recommended reading is the following

• For general background on Category Theory, I'd suggest either Awodey's Category Theory, or Milewski's Category Theory for Programmers. Awodey is targeted for both math and CS people, and I'd recommend reading chapters 1,2 and 3 and skimming chapter 5. Consult at your own leisure.
• Categorical Foundations of Gradient-Based Learning a short paper on applications to ML
• The simple essence of automatic differentiation
• A category theory framework for Bayesian learning

https://discord.gg/eleutherai?event=1321449804009701377

[white obsidian]

Hi! What is the time zone for the meeting?

[sullen estuary]

The time stamps should automatically display in your local timezone

[sullen estuary]

<@&1098008274361663629> The meeting is about to start. I've moved it to another VC to avoid interfering with the other RG

[jovial sand]

A wee bit of intuition - https://www.youtube.com/watch?v=fRJMggrpxRU&list=PLffJUy1BnWj2dBiTZgQ1IDetIQQveRxYO&index=1

Associativity of the binary operation is the necessary condition that ensures the corresponding ternary operation behaves consistently.

Since associativity makes parallel reduction possible, identifying monoids goes hand-in-hand with identifying opportunities for parallelism. e.g. MapReduce etc.

[jovial sand]

Monoids being the simplest associative structure (set + associative binary operation + identity).

[sullen estuary]

You can actually remove the identity to get a semigroup

But yeah, associativity is what you want for parallel computing

[jovial sand]

Neat. This is pretty cool

credit: Tom Leinster

[jovial sand]

operad Δ (finite probability spaces)

[sullen estuary]

https://youtu.be/xMkOx5Gr0rA

[pale orbit]

I entered now to the discord and this reading group is so cool! Is the next session already scheduled?

[sullen estuary]

Soon™️

[sullen estuary]

Alright. <@&1098008274361663629> Sorry for taking so long. Next meeting on <t:1742763600>. We will be talking about the theory of optimization.

The main source will be chapter 5 of Francis Bach's recent book: https://www.di.ens.fr/~fbach/ltfp_book.pdf

To go deeper you can look at the relevant chapters of Boyd and Vandenberghe's Convex Optimization https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf and Bubeck's review https://arxiv.org/abs/1405.4980

Have a nice week!

Now for the less good news. You may have noticed that that the time between sessions has been getting longer. This is in large part because I have much less time for this reading group than I previously did. While I would like for this reading group to continue, it will have to be at a reduced pace unless people volunteer to prepare sessions/take over from me.

[sullen estuary]

https://discord.gg/eleutherai?event=1343210871559557245

forgot to create the event

[sullen estuary]

<@&1098008274361663629> the meeting is starting now

[slow parcel]

@sullen estuary This is the line search in TRPO. Page 14 and 15. https://arxiv.org/pdf/1502.05477

[dusky marsh]

https://arxiv.org/abs/2109.03920

Interpretability is almost like trying to solve the inverse problem of identifying an objective function from observing a model's parameters and outputs, except instead of finding an objective function that describes the cost of incorrect outputs, we're trying to find a utility function that contains concepts humans recognize as familiar.

[dusky marsh]

We're looking to solve the inverse problem, but with the objective function we discover parameterized in a way that's semantically meaningful to humans, so rather than as something like (yhat-y)^2, instead as "this model likes turning static into pictures of dogs".

[sullen estuary]

https://youtu.be/NAio-8To7EU

<@&1098008274361663629> Hey y'all. I finally got around to uploading the last recording. As I said at the end of it, I just don't have enough time or energy to run this reading group at any serious capacity anymore, so this was pretty much the last Math reading group.

It was fun to organize, and I used it as an excuse to learn/brush up on things on more than a few occasions, and I thank everyone who participated in this group.

[tulip gazelle]

Thank you for all your work @sullen estuary ❤️

[true field]

Hey Why dont we have voice channels or video channels

to study together?