santa-2024

[charred fossil]

🎅

[late garden]

Does anyone know why calculator which is given returns that error-no batchsize argument?

[late garden]

In this competition, evaluation codes are strange...From now, all bugs are fixed?

[hardy escarp]

Yeah idk

Eval results also seem to depend on which gpu you use and batch size messes with it

[hybrid depot]

The latest version of metric v28 does not use the parameter batch size anymore. It always uses batch size 1. Earlier versions of metric did use batch size.

[simple badger]

I am looking for a team to join

[hallow raft]

Let's connect, We have some project to discuss AI related

https://www.linkedin.com/in/rohma-refai/

[blissful terrace]

how i can get dataset i have problems while finding the dataset

[low comet]

Hey all, not much action in this channel it seems. I took a stab at the challenge this week with little success. Just getting started, so hopefully better times ahead. Good luck!

[quasi radish]

welcome to santa2024, this is a blackbox optimization problem, so lot of computing power required 🙂

[low comet]

Hey @quasi radish, right, I'm just now thinking that I understand the problem. Took me a moment to get started as I didn't quite understand the ask and was attempting to solve with R, which I was unable to figure out using the Gemma 2 model connection.

I ended up taking a shot with Python, my first venture with the language, still a bit unclear about what we're doing. Ultimately, to operate within the 12 hour time and memory usage limits I ended up splitting the records into groups of 10 tokens, then taking a small sampling (n = 16) to feed into Gemma 2 before taking the best sequence chunks and combining back together. My score isn't great ~ 1089, but I'll take it as this unique challenge is a bit outside my comfort zone. This alone takes ~11 hours to run, so any increase to the sample size hits the time threshold and cancels the script.

I am curious though, how folks are able to generate a single 20/30/40-token length permutation, let alone many, within the limits of the kaggle tool. I assume using the evaluation metric locally helps, but that's just scoring, not permutation generation. Any thoughts?

https://www.kaggle.com/code/josephnehrenz/santa-2024-perplexity-score-minimization

[quasi radish]

I think using external computing resources is a must here, if you want to get anywhere. The search space is inconceivably huge, so any kind of "brute-force" search will unlikely to get you anywhere, therefore, there is a special type of solution that are "better" for each sample.

[quasi radish]

at this point, if you want to do anything better than the top public score, you need to pick your battle, pick a sample, pick the "solution" you think applies to that sample, and optimize the heck out of it, with luck, you can break out of the local minima and find a path downward.

[muted patio]

Hi,
we have a problem in posting new topic inorder to write the write up ( we are the 3rd team)

we can't create new topic ( it says first internal error then too many requests) :/

any ideas !

@analog terrace

[analog terrace]

Have you reached out to support? They're usually really good about helping resolve issues like this.

[thorn crypt]

🎅 🤶