drawing-with-llms

[junior prairie]

I want to answers about key questions for competition as soon as possible.

1. This competition has fatal evaluation metric cons. I can just attacking for SoVit model to get high score. Does this metric NEVER change later?
2. I want to know range which can be changed in this competition. Can I change LLM model? Can I submit with rule-based model? Can I change only text prompt? Can I add arbitrary allowed SVG tag?

[magic sigil]

1. The metric will be changed if organizers finds a need to change it
2. You can do absolutely anything you want(as long as it passes the internal kaggle eval)

[ocean sapphire]

Hi, is it normal that the submission is running for >2 hours? It doesnt give any errors its just still running..

[vast sonnet]

Some of us get a failed submission after waiting for hours: https://www.kaggle.com/competitions/drawing-with-llms/discussion/565052

[trim cliff]

It's a text-to-image generator. Well, how is output evaluated? What happens if my generator has a penchant for Cubist or Abstract Art?

[icy token]

How are submissions evaluated???? I feel as if this is the sort of competition that cannot have a leaderboard, and must be judged by humans, like an art competition, based on certain metrics.

Ok so it looks like the optimal method here is to build around the Sovit model…

[sinful breach]

Yeah this competition sounds like it's begging for adversarial techniques

[icy token]

This competition in a nutshell: https://youtu.be/FoTYV22qZTg?si=018eEMz0gUfsZ4lF

[ocean sapphire]

How are people at >0.29 already its been a day😭

[magic sigil]

yes it is pretty normal

in fact, for most code/(and probably package) comps that is the case

[ocean sapphire]

Why?

[cerulean tapir]

hi who can team with me

please

[cerulean tapir]

please help

[magic sigil]

hello

[cerulean tapir]

can you team with me

[magic sigil]

can you come in dm once?

[quasi solstice]

Dm me bro

[magic sigil]

if you use llms to generate svgs, it takes a lot of time

[quasi solstice]

I'm looking for team if you have any vacant spot lemme know Thanks

[peak crystal]

My submission is getting failed even after running properly and passing the kaggle evaluation test

[jovial trellis]

Check outputs

My scoring is failing and it says invalid submission format but everything seems as per the format
Everything runs smoothly but at the end just after scoring... it shows failed

There are no error messages I can debug 😭

[peak crystal]

that's the point i cant figure out the issue

same model class is working fine in my other notebook

[glad kindle]

Anyone want to collaborate?

[flint gale]

I'm down

[glad kindle]

Ok cool

[hexed turret]

Count me too

[glad kindle]

👩‍🦼

[ocean sapphire]

XD?

[glad kindle]

lolol

[ocean sapphire]

estudias en españa?

[glad kindle]

no

[gray basin]

May I ask what hardware is being used for the evaluation? I would like to estimate the runtime for processing 500 prompts.

[magic sigil]

same as the one you have selected while submitting

you can choose between

1. CPU
2. GPU(1xP100, 32GB VRAM)
3. GPU(2xT4, 16GB + 16GB VRAM)

[hexed turret]

Bro what's exactly different between the two types of GPUs btw 🧐

[magic sigil]

there are some subtle differences

in general use T4

[hexed turret]

So in general t4 better right ?

[magic sigil]

yeah

if you can somehow utilize both the gpus then it is better

[hexed turret]

It's noway possible simultaneously right like cpu + GPU we ve similarly p100 + t4

[magic sigil]

on the benchmark papers, it shows that T4s are way faster for float(FP16) and P100s are slightly faster for double(FP32)

[hexed turret]

And in which cases tpus are better to use ?

[magic sigil]

idk

T4: left, P100: right

[hexed turret]

Ig I never saw tpus being used in comps 🤔

[magic sigil]

there were some tpu specific comps ig

a few years back

[hexed turret]

Oh there seems to be not much emphasis on tpus nowadays

[magic sigil]

¯_(ツ)_/¯

[hexed turret]

Mostly on gpus

[magic sigil]

we dk what google is doing with their own models

maybe TPUs are the key

[hexed turret]

In the dark 😂

[magic sigil]

I don't really know the hardware aspect of everything very well

so I might be saying somethings wrong

[hexed turret]

There r hell lot of varieties of processing units now so can't say a single one for now it's GPUs though

[magic sigil]

we don't really have access to anything else

[hexed turret]

True

[magic sigil]

there is no G in the GPUs we use

even tho they are called GPU ¯_(ツ)_/¯

[hexed turret]

It's due to their origin ig

[magic sigil]

yeah

[hexed turret]

They made it actually for computer graphics

[magic sigil]

T4 does not even support opengl properly

[hexed turret]

It should be called HPU actually (Huang Processing Units)

[hexed turret]

U used opengl in ai projects 🤔

[magic sigil]

no not in AI projects

I used to render stuff in an evolution simulator

[hexed turret]

Oh

[magic sigil]

I second hobby is to make cool graphic things

particularly emergence

[hexed turret]

As cool as ai

[magic sigil]

I like emergent simulations

[hexed turret]

What exactly emergent simulations like though 🤔

[magic sigil]

emergence is complex behavior emerging from simple rules

the most common example I give is the Conway's game of life

[hexed turret]

Btw u r more like Disney research studio researchers they r too at the intersection of cool DNNs + graphics

[magic sigil]

no

I am not really much into the internal graphic aspect

[hexed turret]

Now they even entered robotics which is great

[magic sigil]

I just show stuff on screen, for which I use opengl

I don't really care about what goes on behind the scene

but for AI, I do care what goes on behind the scene

[hexed turret]

U r like Indian Andrej karpathy

[magic sigil]

No

I am like Indian idiot

in fact, I am the one

[hexed turret]

Ok Indian Andrew Ng

[magic sigil]

don't disrespect them

they are big names

[hexed turret]

Btw I know one Indian from WB who's doing great in ai

Today

[magic sigil]

let us move to dms or smth

[trim cliff]

this counts as advancing the field of text2image generation 🤣

and I still don't get why you'd want to use an LLM to draw SVG code based images. Maybe a system that renders faster on mid ressources than Stable Diffusion Models would make more sense. About the only thing I'd use from LLM is word tokenizers, like Bert.

[sinful breach]

2.5 pro:

😍

[violet quarry]

Does drawing sketches with LLMs gives sequence of strokes or they outputs a direct images?

[smoky flower]

can see, AI is in its childhood 😁

[sinful breach]

In this case it gives the SVG code, so I guess a bit more like sequences of strokes indeed than the standard image generation.

[trim cliff]

nailed the eyes

[stray aspen]

Hi, just wanted to check explicitely that the use of closed-weights models like Sonnet 3.7 is allowed, and that the cost to run it is all on ourself?

[hexed turret]

How can u use closed weights model anyways ?

[stray aspen]

What do you mean 🤔

[sinful breach]

Your notebook submission doesn't have access to the internet at test-time, so it won't be able to use models like Claude.

However, you can use Claude to generate a dataset for an offline model that you can then upload to kaggle and run in your notebook.

[stray aspen]

Ah that makes sense, thanks!

[jovial trellis]

Does anyone here have a script for fine-tuning flux or stable diffusion? Would love to have a look

[steep fossil]

Is stable diffusion allowed? I see it in a lot of the solutions and just wanted to confirm.

I think Stable Diffusion is a latent diffusion model and not technically an LLM, and seeing as the title is "Drawing with LLMs", I wanted to confirm.

SD -> png -> svg is a little different from LLM -> svg so I wanted to ask for clarification whether that is in the spirit of the rules.

[ocean sapphire]

Yes its allowed

[hazy carbon]

There are full fine tuning and LoRa scripts in Huggingface examples directory

Has anyone tried the star-vector model on the test set here? If no one has done it yet, I could write a script today, just wanted to see what kind of scores it is getting

[jovial trellis]

tried it but couldn't get it running, there are some bugs in the repo

[jovial trellis]

yes was trying that only, thanks

[hazy carbon]

Yeah just checked, they haven't released txt2svg model yet, only img2svg

Even img2svg their gradio demo in the repo has bugs, had to use hf space to try it out

[untold flower]

I think it won't work because FlashAttention2 doesn't support Kaggle's P100 GPU and T4 GPU. Here is my noteboook: https://www.kaggle.com/code/baorbaor/test-starvector-model
I hope everyone will give me some feedback ❤️

[hazy carbon]

Can just disable flash attention and use normal attention

[hazy carbon]

Also what's your plan for this? Because the authors have only released img2svg rn, and the task is txt2svg

[untold flower]

sorry i forgot :<

[steep fossil]

https://omnisvg.github.io

pretty interesting SVG gen paper just released

[hazy carbon]

Haven't read the paper yet, but I am happy we have now atleast 2 high quality large scale svg dataset. Just 2 months ago, when I was looking at this problem I couldn't find a single dataset that passed the vibe check

[trim cliff]

Checked it out. Impressive. Kind of just deaded the whole Competitions raison d'être.

[lethal hound]

Why can’t I see any data inside the test.csv file on the competition page? Am I missing something?

[balmy ivy]

Hello everyone, wondering is there anybody still facing "Package Prediction Error"？This link from discussion (https://www.kaggle.com/competitions/drawing-with-llms/discussion/565052#3182254) shows that it's actually a kaggle bug, but it has been two months... I have no idea wether it's kaggle's problem or my problem👀

[upbeat jewel]

Thanks for the answer, but did you actually get confirmation from the competition organizers? I want to be 100% sure exploring SD

[ocean sapphire]

Look at all the public notebooks with SD, I didnt get personal confirmation but why wouldnt it be allowed?

[upbeat jewel]

Well, the rules don't explicitly allow/disallow the use of diffusion midels and many people are using them but the competition's name is "Drawing with LLMs" so it would be really reassuring to get confirmation from the host.

[clear yarrow]

Check if your output is satisfying the SVG constraints (the link to that present in the sample notebook), since I was getting prediction format error because of that

[upbeat jewel]

https://www.kaggle.com/competitions/drawing-with-llms/discussion/564984#3134500

[ocean sapphire]

What are you trying to say? SD is allowed

[upbeat jewel]

Yes it is! And that is a confirmation from kaggle staff

[dense helm]

ModuleNotFoundError: No module named 'kaggle_evaluation' i am getting this error does anyobody know what to do about it

[balmy ivy]

Thank you！I've made it

[balmy ivy]

import kaggle_evaluation

[dense helm]

i did import it but it still gives error

[fallen dust]

Hi @woeful basalt, Jeff here

[trim cliff]

Potential future Video applications of "drawing-with-llms", 🤣 : https://youtu.be/-M7-Sew5aU8

[trim cliff]

Diffusion models don't produce SVG code for image generation which is key to the competition, as I understand it.

[upbeat jewel]

Diffusion Models produces raster images but you can convert raster to svg code.

[trim cliff]

I am aware. Someone posted a link to a project doing something similar to this, here: #drawing-with-llms message

go for it though

SVG is basically turning images into a set of instructions for rendering VS values for pixels. In the early days of the Net, it meant less info had to be sent for creating an image, client-side, aka, not sending thousands of pixel values but instructions for constructing an image. In this day and age though, not as much necessary. IMHO. It was a creative compression algo, of sorts, for its Time.

[rugged eagle]

I am interested in becoming a member of teams which require team members.

Feel free to shoot me a DM.

[safe silo]

What are the reasons for getting package prediction error? and how to debug it?

[rancid token]

a lighthouse overlooking the ocean

[trim cliff]

It looks like a headshot of Krusty the Clown 🤣

[rancid token]

https://media.istockphoto.com/id/1398712599/photo/lighthouse-on-smal-island-in-the-sea-at-sunny-day-in-summer-aerial-top-view-of-beautiful.jpg?s=1024x1024&w=is&k=20&c=VRtQRONzmT8I_FOnuZyXgHQShj4fC9gPiAf5sT60pZU=

[small notch]

Hello, black-forest-labs/FLUX.1-dev has a flux-1-dev-non-commercial-license. Can this be considered to be within the bounds of the rules for this competition? It would be really nice if a Kaggle moderator could confirm this 🙂

[clear yarrow]

check if your generated svg is satisfying the SVG constraints (the link to that is present in the sample notebook)

[livid osprey]

Hi, did anyone face Kaggle Errors while submitting in this competition? Ive had cases where my submission runs for a couple houurs and throws a Kaggle Error, if I try rerunning, it fails in about half an hour, Ive also had a case where my subission ran for over 15 hrs and threw a Kaggle Error. Ive had this 4 times now and not quite sure how to resolve this as the docs on this is very limited and it asks you to simply retry.

Kaggle Error: A rare system error. Please try resubmitting to resolve the error and contact Kaggle Support if it persists.

Any help regarding this is much appreciated! 🙏

[clear yarrow]

I am getting package import error for some reason even if I haven't imported any new package compared to a previous version that didn't run into this error.

[safe silo]

How to save a model fine tuned by me so that it can be used by the evaluation system?
Should I just save it in /kaggle/input or do I have to do something else?

[waxen valley]

Save in kaggle/output

Then when the notebook has saved, go to outputs and convert to a dataset or model

[zenith geyser]

I'm getting dependency installation error when running notebook
Ive added pip installs in Add-ons -> Install dependencies.
And internet is on
Yet notebook is giving import errors
How can I resolve this?

[random lotus]

Is anyone still facing Package Prediction Error when submitting ?

Our team ran into that yesterday...thought it's fixed 😐

[zenith geyser]

I was able to submit notebook successfully, but while scoring it failed.
It shows notebook threw an exception. I've checked output and logs but I couldn't find any error.

How can I know where the issue is

[hazy carbon]

How are you all saving your model for submission? I tried saving it in package asset but when I save the version it says I ran out of disk space

[trim cliff]

Someone show their Drawings. Let's see them SVG Unicorns, etc.

[gloomy egret]

Hola 👋