data-science-and-ml

[final cobalt]

https://hastebin.com/share/oxerukoqex.py

My algorithm for sorting all pokemon cards by frame XD

I'll need to do some manual sorting, particularly with respect to splitting standard and full-art cards

But this does most of the leg work

[rich moth]

It ran for 5 epochs but I got some crazy tkinter error, but I'm not even using it lol

[final cobalt]

Maybe tkinter is used by whatever tool you're using to build the image files after reconstruction?

I doubt any of the ML/NN frameworks use it

[rich moth]

ya, your probably right. I think I fixed it once before enabling tkinter. The sorting algo looking nice

[final cobalt]

@rich moth It's a KNITTED ZUBAT!!!!!!

[lapis sequoia]

Helu

Can anyone help BELUGA?

[final cobalt]

That depends what BELUGA needs XD

So long, and thanks for all the fish?

[lapis sequoia]

Helu can anyone provide the resource for nlp?

[final cobalt]

XD

I can do you one better

[lapis sequoia]

Do u know who beluga is

Go and search on yt

Btw i am 10000000th copy of beluga

[final cobalt]

https://chatgpt.com/

No matter what anyone else tells you, ChatGTP is a fantastic resource for learning technical skills. It'll answer any question, never lose patience, and it'll be as simple or detailed as you need it to be

Just remember that it can hallucinate

[lapis sequoia]

Whatt but i have not premium

[final cobalt]

Get it

[lapis sequoia]

I have no money

[final cobalt]

It's only 20 bucks a month, and it's 20 bucks well spent I assure you

[lapis sequoia]

I have 0

[final cobalt]

https://tenor.com/view/dean-winchester-supernatural-relatable-relatablemoods-gif-7719768255809338997

[lapis sequoia]

I want to earn money due to that i am learning skill but nowadays everything is premium

[final cobalt]

Well, the sad news

Is that if you're here asking for advice about learning NLP/ML

Then you're at least a year if not two from being able to monetize your skills

If you want quick money, try data entry

Or data annotation

[quaint mulch]

I think doing it 5 times is a good baseline to try

Is this like a learning / hobby project so you are purposedly not using existing pre-trained models? or fine-tuning those?

This is more towards ML than data science: https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf

https://github.com/EleutherAI/cookbook

[grand breach]

any point in imputing a categorical variable that has 47% of values as -1 ? will i lose critical information on dropping it ?

[narrow finch]

Hey I am a beginner and I want to be a ai developer

Can anyone guide me

[jaunty helm]

depends
what's the variable about? how much is missing? etc

[jaunty helm]

see pinned

[narrow finch]

What

[jaunty helm]

see the pinned messages in this channel

[neat siren]

hello

anyone experienced in data extraction using beautifulsoup

[grand breach]

well it's an anonymized variable (business reasons), it's not missing but anamoly or outlier values, having 47% of -1's

[jaunty helm]

so 47% is -1, 53% are others?
imo that's still pretty informative so keep it

[grand breach]

yeah even i think the same, how should i impute it ? replace with mode ?

it has many classes maybe some 250

[quaint mulch]

maybe just don't impute it then?

[jaunty helm]

"imputing" would imply that you have missing data
you just said you don't have missing data

[grand breach]

sorry my bad

[grand breach]

but those outliers would cause a problem then ?

[jaunty helm]

you don't have to apologize
you just said conflicting things (you don't have missing data & are imputing) so I'm confused on how to help you

[grand breach]

then what to do ?

[quaint mulch]

missing data might cause a problem.
But if your dataset is large and IID enough, the worst case scenario is that the feature will get ignored

But I generally think that, a bad imputation could cause an even bigger problem.

[neat siren]

hello

[grand breach]

there's a correction, it's not missing data it is filled with -1 (an invalid) which might add noise to the model, should I replace them with mode or there is any other technique to replace with some synthetic data

[quaint mulch]

this feature is a categorical feature and you are doing 1 hot right?
you can make a new category called invalid ?

[neat siren]

@storm valve see in print data the output doesn't include movie names cz it has embedded link , how to get the names then

[storm valve]

#❓｜how-to-get-help

[onyx frigate]

Hey guys do you know about ai agents.

So should we learn how to make them using code or just by the no code platforms.

What's the difference between these 2 approach

[grand breach]

not one hot but label encoding as there are too many groups

[onyx frigate]

Have any one of you made an ai agent?

[quaint mulch]

depends on what you want to achieve.

[onyx frigate]

Like let's say I'll start by creating an ai bot to manage my LinkedIn dms.

But later on I'll create big projects

[quaint mulch]

same idea, just make a new category?

[quaint mulch]

what counts as an AI agent?

[quaint mulch]

generally, always start with no code, see if it is good enough.

generally, always start from the easiest, cheapest, and fastest, and see if it is good enough

[onyx frigate]

I think you're right

[grand breach]

I don't know what's wrong with Kaggle, My session just crashes when I try to run pearson correlation on my data set, I have even tried sampling my data set but that didn't work.

[onyx frigate]

Also i wanted to train and mess around with flux lora model but I don't have a gpu are there any free online alternatives

[quaint mulch]

google colab

[final cobalt]

Yes, and, I have a strict ethical critera

This is all building towards a proper diffusion model which is trained on public domain, creative commons, commercial, and synthetic data only

[quaint mulch]

yea, I'm actually equally curious. How come chatGPT generate image of any resolution/aspect ratio?

[inner creek]

Even though it's not efficient and resourceful. What if they just generate a full img but then cut them to ratio

Certainly easier to train that way

[final cobalt]

There's also tiled diffusion and adaptive layers to handle resizing

The latter being a rather yucky approach

[unkempt wigeon]

But that begs the question is there a way of detecting if a neural network is a deep learning or just a regular neural network

[charred egret]

Neural network is the underlying idea that is used in deep learning so i’m confused about what you’re asking. neural network with 1 hidden layer (shallow) and another with 1 billion hidden layers (deep) both use the idea of neural networks

[unkempt wigeon]

I know but, if somebody can make a virus that can differentiate between a a regular neural network in a deep learning model and insert poison data and it retrains the network

[unkempt wigeon]

But is there a way for a computer virus or a program to figure out how many layers there are in a neural network sorry

[charred egret]

it depends on the virus and what it can do and what vulnerabilities it can exploit. if the whole system is compromised, the attacker can do anything. but in that scenario the vulnerability still won’t be in the neural network.

honestly look up how neural networks learn. it’s just mathematics, you can’t make it suddenly replicate virus and spread it just because there are some wrong data in the training set or even if you manually tweaked its neurons. at the end of the day it’s just doing a bunch of matrix multiplications to put it very very simply.

[unkempt wigeon]

Could deep neural network learn to be like a person just by using data let's say if I took some data video on somebody that I knew and the network kind of training could it act like the person their mannerisms etc by accident if you don't even program it in sorry

[serene scaffold]

No.

[unkempt wigeon]

How so

If you think about it a neural network and a human brain are just complex math figures although one is biological one is mathematical but in essence they're just a computer that can crunch Mass so if you give certain mannerisms or videos of I account information you can make a network that can act like the person but not have everything there like how normal human would act sorry

[serene scaffold]

The way that you talk about neural networks (not just right now, but in general) indicates to me that you don't understand what they are or how their training relates to the task that they perform.

If you want to learn more about AI, I suggest you start by learning concepts that are more approachable to beginners and work your way up. Your thought experiments really don't make any sense.

[serene scaffold]

A neural network is a set of numbers and an associated computation graph. What would the output of a neural network be that would be an emulation of a person's mannerisms?

Would it be text that a person might say, given some amount of hypothetical text said by others in a conversation?

Would it be a deep fake video of that person?

I sounds to me--and I might be misunderstanding you--that you expect to somehow produce an entire behavioral model (whatever that might mean) of a person given video footage of that person.

[unkempt wigeon]

Why mean is a neural network that has a video of a person acting out their normal day from the person's perspective with a microphone that it looked into a mirror would be hard to see if neural network takes all this data crunches it it could make a personality when you have to take most of the personality from the person that it's being trained off of

[faint quail]

bro this model has my gpu looking like a sound visualizer

[final cobalt]

Have you ever tried not speaking in run on sentences?

https://tenor.com/view/english-motherfucker-speak-do-you-speak-it-sam-l-jackson-gif-4693590

[serene scaffold]

!rule 4

[arctic wedge]

Rules

4. Use English to the best of your ability. Be polite if someone speaks English imperfectly.

[final cobalt]

XD Sorry - not meaning to be rude

Trying to be funny

Failing, apparently

[serene scaffold]

what you said actually sounds very rude. please be more thoughtful in the future.

[unkempt wigeon]

No it's ok if someone hit me square on my head with a wooden base ball bat I would not care sorry

[serene scaffold]

@unkempt wigeon have you considered following along with a book about neural networks?

[unkempt wigeon]

Yes

[unkempt wigeon]

It's okay you can be rude to me all you like. I don't mind

[serene scaffold]

it's nice that you're forgiving, but you can't grant people permission to break the #rules as they pertain to you.

[unkempt wigeon]

What I mean is I don't mind Now if it was someone else then yes but me you can be be ruder than anything and you could punch and scream at me I wouldn't care

My apologies

[serene scaffold]

@unkempt wigeon what neural network book are you following?

[final cobalt]

Honestly - I wasn't trying to be rude. Mostly, I just wanted an excuse to post that GIF from Pulp Fiction because it's funny

[serene scaffold]

I believe you weren't trying to be rude.
let's move on from this.

[unkempt wigeon]

I I know this isn't concerned a book but I do because it's electronic I can access it from anywhere and if a computer goes down

And or a hard drive or if I misplace the book it's not like I'm wasting any money sorry:

https://www.w3schools.com/python/python_ml_getting_started.asp

@final cobalt I believe you weren't being rude

[unkempt wigeon]

I know it's not concerned a book but it is very detailed from when I can read sorry

[serene scaffold]

it doesn't need to be a "book" as long as it's "feature length".

[unkempt wigeon]

Feature len?

[serene scaffold]

if you go to a movie theater, they'll show you a few previews that are a few minutes each, and then a movie that's about two hours. and the movie is the "feature".

[final cobalt]

Personally, I do the majority of my learning through ChatGTP

Which I know, I know, everyone says is stupid

But it has more than enough juice to get you conversant in a subject. You just need to take everything it says with a grain of salt, and use it solely as a rubber ducky / readings-collator

[serene scaffold]

I do the majority of my learning through ChatGPT
and
[I] use it soley as a rubber ducky/readings-collator
these statements appear to be in conflict? @final cobalt

[final cobalt]

As a readings collator, it can gather information for disparate sources and demonstrate them to you whilst being interactive

It goes and finds the readings for you, and you can ask it follow up questions. You just have to remember that it's a people pleaser and it's also dumb as a twig

So don't ask it to interpret, just ask it straightforward questions

[unkempt wigeon]

So that's the training time?

[mystic peak]

would it be possible to make a machine learning program for fighting games

[unkempt wigeon]

I want type of network were you thinking genetic or basic convolution because if it's a threening game you would have to use an open AI background but if you make it yourself then you can calculate what might go on plus 2D physics are easier to handle on a computer dependent on your GPU or CPUs usage

[rich moth]

absolutely. maybe you can get some ideas from https://farama.org/projects

a good start is maybe find a nintendo old school game like kung-fu

[rich moth]

Im not sure if you mean something like street fighter or tekken style.

[unkempt wigeon]

How did you make your capture the flag did you make the network an import?

[rich moth]

I used this ```
class QNetwork(nn.Module):
def init(self, input_dim, output_dim):
super(QNetwork, self).init()
self.fc1 = nn.Linear(input_dim, 128)
self.fc2 = nn.Linear(128, 128)
self.fc3 = nn.Linear(128, output_dim)

def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    return self.fc3(x)```

and some RL learning

attention too.

[unkempt wigeon]

Well really was trying to mean is did you come to game environment yourself sorry

[rich moth]

Oh, the idea of CTF?

I think I understand. I designed it myself. I made it with just these imports import pygame import random import numpy as np import torch import torch.nn as nn import torch.nn.functional as F

[unkempt wigeon]

CTF?

[rich moth]

Capture the flag.

[unkempt wigeon]

Sorry is the neural network plugged into the game itself or could you just import it and everything gets saved on the import sorry and I see you've used the class for you now Network sorry

[rich moth]

The Qnet is intergrated into the game enviorment. The players have a q network for decison making and it trains while the game runns. I have the player parameters saved so it loads and uses the network.

[mystic peak]

would you count the newest street fighter game as 2d or 3d because it has 3d models but it plays in a 2d plane

[unkempt wigeon]

Why was hoping someone I make my own network is make it where I can export it or import it into a game it would learn then make changes to the main python file although it probably would be a good idea in just to import it not change the network on the original already I've never done this and I'm trying my best to learn

[rich moth]

never thought about it lol

So a generic qnet model that you can swap between different games that automatically adapts and updates the internal code?

[unkempt wigeon]

?

[unkempt wigeon]

Do I need to have a class or anything for a girl Network well unless it's playing for games but anything else sorry

[desert oar]

you might want to work through some actual material about deep learning

examples: https://d2l.ai/ or https://www.fast.ai/

[rich moth]

There's some cool stuff on there @desert oar I found this on there too. https://www.answer.ai/ The WebGPU was a cool read.

[remote stream]

anyone knows how to annotate video or a free software to reduce my stress

pls reply someone

[grand breach]

should i drop highly correlated features for training with linear models or keep them for training and later apply regularization techniques ?

[quaint mulch]

You can do PCA, or other similar things like that.

[grand breach]

i've read that RF or DT are immune to multicollinearity or redundancy

[quaint mulch]

Well yea, it also kinda depends on what model do you use.

[unkempt wigeon]

Which is the best type of network sorry

[charred egret]

There’s no such thing. Different networks are suited for different types of problems. It’d actually even be wise to consider if you even NEED to do deep learning

[unkempt wigeon]

I want to do deep learning because if you kind of do the hard stuff first the easiest stuff is beyond easy and is there a way of combining two networks

[serene scaffold]

You should not start with deep learning.

[unkempt wigeon]

How so?

[serene scaffold]

If you start with the hardest thing, you won't understand what you're doing and will give up before you accomplish or learn anything.

[charred egret]

It can be counterproductive if you don’t have the prerequisites. You’ll encounter too many roadblocks that can be discouraging because you don’t know what’s happening. You’ll end up spending most of your time on the non-deep learning things because otherwise you wouldn’t understand what you’re doing.

[unkempt wigeon]

I know about numerical values and I know how to turn images into a raise of numb I can crunch therefore giving it some sort of vision my apologies

[pearl parrot]

Finished Python OOP, jumped into the book Python Data Science Handbook by Sebastian Raschka. Will I be fine? I’m nervous

[fallow coyote]

I learnt this the hard way. If you want to get into the whole machine learning AI space, you must have a good grasp on the mathematics behind it. If you don't you wont be able to fully utilise all the ML libraries available. Atm, Im building my programming skills around ML (e.g. databases and data analysis). Im at university where in the first year well be going over the general mathematics. Honestly, this area of computing requires you to go to uni to learn this stuff as its a whole other level of complexity

[unkempt wigeon]

How many neurons are I a shallow Net?

!e

import numpy as np

X = np.array([[1,2,3,4,5],
])

W = np.array([[1,2,3,4,5],
])

B = np.array([[1,2,3,4,5],
])

Output = np.dot(X,W) + B

Print (Output)

[arctic wedge]

:x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 12, in <module>
003 |     Output = np.dot(X,W) + B
004 |              ^^^^^^^^^^^
005 | ValueError: shapes (1,5) and (1,5) not aligned: 5 (dim 1) != 1 (dim 0)

[unkempt wigeon]

What counts as a shower Network or a deep Network sorry

[fallow coyote]

no idea. apologies. make a help thread on the python-help channel. should get a quick response

[hybrid acorn]

:Error during chat completion generation: '<=' not supported between instances of 'method' and 'int'
BUT I DON'T use <= in that context QQQQQQQQ

[unkempt wigeon]

done

[unkempt wigeon]

im sorry im just starting

[fallow coyote]

Dont worry. We all are. Maybe in 2 years Ill be able to help you XD

Otherwise, ask the others who are multitudes smarter than me on this discord

[unkempt wigeon]

To become a master at an r you have to do it multiple times so if you do three neural networks per day you can become a master maybe half a year maybe sorry I'm trying to do the math because some people who make noodles if you make noodles more than a couple times a day you learn faster sorry

[serene scaffold]

You don't "do three neural networks a day".

If you're serious about machine learning, please follow along closely with a specific resource like a book or course. The approach you are trying to take is not going to work

[hybrid acorn]

oh, I found it, named args helped

[desert oar]

I already suggested some structured learning material you can use. yes daily practice will help, but you really ought to start working through some actual structured learning materials at this point

[tulip wyvern]

If my KNN outperformed my LightGBM model substancially (33% - 5%), is it likely that i made a mistake in my code or does KNN just outperform gradient boosting models on some tasks?

[desert oar]

what task? what kinds of features? how much data?

how did you evaluate?

it's always possible

[tulip wyvern]

are these rhetorical questions

[tulip wyvern]

okok ic ty

just didnt know the difference could be that big

cuz i had the (incorrection) notion that lightgbm performs well on all tasks

[desert oar]

No, they are real questions

I don't have any specific scenario in mind, but you have to consider all of those things when you are thinking about model performance and what works well

There might be something unusual and specific about your task where nearest neighbors is actually better than global curve fitting

That or your code or training pipeline is bad somehow

But it's never about generalities, it always comes down to the specifics of your problem and the way you set it up

[tulip wyvern]

o i see
im trying to predict the leading pokemon of a user in pokemon showdown given both side's full team (so each feature is a categorical variable representing the name of a pokemon) and i have like 50k entries

[tulip wyvern]

yeah im worried that my training pipeline is broken

[hybrid acorn]

can you run multiple models with llama_cpp? any pitfalls or should I use something other langchain processing

[rich moth]

you can run multiple models, but llama models are pretty resource heavy. try it out.

it gonna boil down to your system and the requirments of the model.

[finite thicket]

[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/d/Projects/sync/get-dissed/get-dissed-prototyping/pixtral_test.py", line 17, in <module>
[rank0]:     llm = LLM(model = model_name, tokenizer_mode="mistral", trust_remote_code=True)
[rank0]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 214, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 564, in from_engine_args
[rank0]:     engine = cls(
[rank0]:              ^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 325, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:                           ^^^^^^^^^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 47, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/executor/gpu_executor.py", line 40, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/worker/worker.py", line 183, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1016, in load_model
[rank0]:     self.model = get_model(model_config=self.model_config,
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 19, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 403, in load_model
[rank0]:     model.load_weights(self._get_all_weights(model_config, model))
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/models/pixtral.py", line 259, in load_weights
[rank0]:     self.language_model.load_weights(llm_weights)
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/models/llama.py", line 493, in load_weights
[rank0]:     for name, loaded_weight in weights:
[rank0]:                                ^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 378, in _get_all_weights
[rank0]:     yield from self._get_weights_iterator(primary_weights)
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 364, in <genexpr>
[rank0]:     for (name, tensor) in weights_iterator)
[rank0]:                           ^^^^^^^^^^^^^^^^
[rank0]:   File "/home/zghan/.local/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 406, in safetensors_weights_iterator
[rank0]:     with safe_open(st_file, framework="pt") as f:
[rank0]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: unable to mmap 25365548952 bytes from file </home/zghan/.cache/huggingface/hub/models--mistralai--Pixtral-12B-2409/snapshots/df119bf36c0cedc6ffdc9ca6c58ebf51f9771ef7/consolidated.safetensors>: Cannot allocate memory (12)

from vllm import LLM
from vllm.sampling_params import SamplingParams
from huggingface_hub import login, whoami

# Authenticate with Hugging Face only if not already logged in
try:
    whoami()
except Exception:
    print("Not logged in. Please enter your Hugging Face token.")
    login()

# https://huggingface.co/mistralai/Pixtral-12B-2409
model_name = "mistralai/Pixtral-12B-2409"

sampling_params = SamplingParams(max_tokens=8192)

llm = LLM(model = model_name, tokenizer_mode="mistral", trust_remote_code=True)

anyone know whats going on here?

[verbal venture]

let's say I want to make certain decisions that are connected to one another, where each decision has a path of its own (that leads to other decisions), and each comes with a reward but also a consequence. I am trying to determine the least negative decision to choose. which algo is best for that?

[untold fable]

i got a free corse on corsera for free

[unkempt apex]

minimax ? or decision tree ?

both would work I guess

[rich moth]

Looks might its too big for your system memory requirments. Try the 7B model see if that works.

[finite thicket]

im pretty sure i do

i have 32gb of ram

[rich moth]

for a 12B model?

humor me, see if the 7B works 🙂

[finite thicket]

ill try

Traceback (most recent call last):
  File "/mnt/d/Projects/sync/get-dissed/get-dissed-prototyping/pixtral_test.py", line 17, in <module>
    llm = LLM(model = model_name, tokenizer_mode="mistral", trust_remote_code=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zghan/.local/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 214, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zghan/.local/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 561, in from_engine_args
    engine_config = engine_args.create_engine_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zghan/.local/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 874, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zghan/.local/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 811, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/home/zghan/.local/lib/python3.12/site-packages/vllm/config.py", line 183, in __init__
    self.hf_config = get_config(self.model, trust_remote_code, revision,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zghan/.local/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 141, in get_config
    raise ValueError(f"No supported config format found in {model}")
ValueError: No supported config format found in mistralai/Pixtral-7B-2409

not sure what this means

[finite thicket]

whats the difference between 12 and 7b anyways

[rich moth]

well its 7billion parameters compared to 12billion so testing to see if it is indeed memory related. But its a size difference , im just guessing you cant fit a 12B model. You need like 25gigs to load it but we have to consider your overhead. Are you in windows?

let me see

[finite thicket]

yes, using wsl

[rich moth]

here ill just try your code

ok its loading

INFO 10-06 22:54:48 model_runner.py:1025] Loading model weights took 23.6552 GB

There ya go 🙂

[finite thicket]

73gb holy shit

is that normal?

[rich moth]

well i have a lot of stuff going but my wsl env is 33gigs

[finite thicket]

is this exclusive to wsl? would it be less on smthn like a mac

[rich moth]

on straight linux it might work

[finite thicket]

im getting an error on my mac

RuntimeError: Failed to infer device type

[rich moth]

ok i rebooted to get some fresh readouts. Before I started it my overhead was 31gigs, its leveled out at 76 gigs for ram, it seems. So 45 gigs to loaded it all up?

im not sure about your mac.

[finite thicket]

ah got it

im new to all this stuff, didnt know it takes up that much resources

[rich moth]

dont worry bout it, we all start somewhere. it just seemed like a memory issue from past expereince.

[final cobalt]

Anyone have any advice for creating consistent characters using Stable Diffusion?

I've got a few OCs I'd like to train LoRA for, and I can get pretty close using the tag "character sheet" and the characterturner embedding

But not quite close enough that it's the same character every time. I've developed a workflow for getting really really close, but it's painstaking

[rich moth]

I've been playing around with it. Try this llm = LLM( model="mistralai/Pixtral-12B-2409", tokenizer_mode="mistral", trust_remote_code=True, gpu_memory_utilization=0.9, swap_space=4, # GB cpu_offload_gb=4, # Offload 4GB to CPU max_seq_len_to_capture=4096, # Smaller sequence length to save memory dtype="float16" # Use mixed precision to save memory )

only requires 19.5 gigs for the weights

[finite thicket]

i'll give it a go

[finite thicket]

RuntimeError: unable to mmap 25365548952 bytes from file </home/zghan/.cache/huggingface/hub/models--mistralai--Pixtral-12B-2409/snapshots/df119bf36c0cedc6ffdc9ca6c58ebf51f9771ef7/consolidated.safetensors>: Cannot allocate memory (12)

i think i just need a better system lmao

my cpu isn't exactly the strongest, its an i5-12400F

[jaunty helm]

as a rough estimate, a full precision model probably uses bf16 to store its weights, so 2 bytes per parameter
so a full precision 12b model would take ~24gb of memory to store its weights, + some more to store the context
however the library might be trying to fit all of that onto your gpu, that'd mean you need 24gb VRAM and not system ram

to use your sys ram instead you'd need to offload to cpu
Plunder seems to have alr showed how above, though in that code it's only offloading 4gb to sys ram, considering the full 12b model then you still need 20gb vram; that's still a rtx 4090 for reference (4090 has 24gb vram, 4080 has 16gb)

[lapis sequoia]

hello guys, I recently completed my Bachelor's degree in Computer Science and I'm gonna take admission in MS DATA SCIENCE. I'm a Python programmer but a beginner. So, can you guys give me road map or is here any at the same level so we can learn together?

[onyx frigate]

Start with the basics like take a look at those python in 12hrs videos and try to do as many mini projects as possible.

Also you should be clear what's your objective that you're learning python for.

Do a research on what are the most crucial topics for what ever you want to do try to focus more on that topic.

Don't try to perfect everything just skim through because no one can learn complete python just focus more on imp topics.

[finite thicket]

Ah that makes sense, ty

[unkempt apex]

bro we have now 1B also

for llama

but not for mixtral

[tawdry monolith]

Does 3 blue 1 brown playlist essence of linear algebra,calcus covers what I need for ml?

[jaunty helm]

Qwen2.5 0.5b:
tbf it's borderline unusable

[final cobalt]

My school's comp sci club is having a t-shirt contest and so last night in the wee hours of the morning I broke out the old Stable Diffusion to see what I could see. Our unofficial mascot is a rabbit, so I thought I'd run with it.

[unkempt apex]

yeah I used qwen actually 1B one, it needs to be fine-tunned first

[fierce plank]

can you recommend some good and FOSS tts models?

[idle swift]

Chat how hard is gym open ai for someone in grade 12?

[fierce plank]

depends on your experience but if you've got some with ai it shouldn't be that hard

[unkempt wigeon]

Doesn't anyone know if there's a YouTube compendium for everything that you need to know for neural networks the mathematics etc sorry

[serene scaffold]

Start with 3blue1brown's video series

[left tartan]

I hope it's a glow in the dark shirt.

[unkempt wigeon]

Thank you

[final cobalt]

I got shot down flat

No one was interested in letting AI be included

That said, I might get one or two of these on a hoodie just for myself

The first and last ones, probably

[rich moth]

I wanted to get this hoodie made lol

Tell me that ain't dope! I dare someone!

[final cobalt]

Tis quite dope

[hybrid tangle]

any library recommendations for data vis outside of seaborn that anyone recommends?

[unkempt wigeon]

Which videos should I use?

[desert oar]

i prefer plain matplotlib over seaborn most of the time. if you want to try something completely different you can try holoviz, but nothing is nearly as polished or well documented as mpl

[onyx frigate]

This is sick

[midnight nacelle]

Neural Network algorithms

He has videos on it

[unkempt wigeon]

Thank you

[midnight nacelle]

Talks about backpropagation, and i think different learning algorithms

[final cobalt]

So lemme run a problem past you all

[arctic wedge]

:incoming_envelope: :ok_hand: applied timeout to @final cobalt until <t:1728353332:f> (10 minutes) (reason: attachments spam - sent 10 attachments).

The <@&831776746206265384> have been alerted for review.

[serene scaffold]

@final cobalt use the paste bin

!unmute 1194743800556441621

[arctic wedge]

:incoming_envelope: :ok_hand: pardoned infraction timeout for @final cobalt.

[final cobalt]

XD Sorry, was NOT spamming. Not trying to anyway. Was gonna show y'all some of the pokemon cards I'm trying to build a network to process

For context. Sent too many though (10)

[serene scaffold]

in what way will it process them?

[final cobalt]

Sorry, was having a bath

[eager verge]

How do you recommend someone to learn the whole ai, machine learning, llm and whatever there is to have a simple broader picture of it all? Just enough to make dialogue with someone experienced!

[final cobalt]

I'd like to build a network to automatically learn and apply segmentation masks to pokemon cards. I've separated them all by frame configuration (since the shape of cards has changed many times over the years) and I figure I can use contrastive learning to force a convolutional network to focus on what's consistent across images

In theory, the frame's stay the same but the image changes. Simple

Except the text on the card also changes. Makes things much harder.

I guess I just wanted to ask

What would be the standard approach to having an NN learn on it's own to separate card frames from card images?

[serene scaffold]

expect this process to take years.
start small and work your way up. be humble about what you know, and stay curious.

and you'll never learn the whole. there isn't enough time.

[unkempt wigeon]

Is this the right one?
https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&si=vWmCk1AeUmchdlk4

[serene scaffold]

yes

[midnight nacelle]

Yes

[serene scaffold]

yEs

[midnight nacelle]

yeS

[serene scaffold]

no
that's not the trifecta

should be yeS

[unkempt wigeon]

Thank you

Should I listen to the linear algebra section too?

[final cobalt]

My linear algebra teacher had a very, very thick Ukrainean accent

[midnight nacelle]

Yes

[final cobalt]

And spoke in broken English

Not that there's anything wrong with that, but it definitely made things a little harder than it might have otherwise been XD

[midnight nacelle]

nvm

[final cobalt]

Very, very much so

[serene scaffold]

a fundamental flaw in the university system is that subject matter experts are not necessarily the best at disseminating their knowledge.

[final cobalt]

To make matters worse, there was a girl in the class who was very, very autistic. Nearly the can't-make-eyecontact kind. Again - not that there is anything wrong this this. In fact, I found her kinda inspiring, but

When she was nervous or frustrated, she was incapable of keeping herself from talking

So one the one hand there was a teacher who already had difficulty explaining a very complex subject. On the other, a classmate who couldn't keep herself from talking at all times even when the teacher was teaching or we were doing an exam

Twas a difficult class

[serene scaffold]

one would expect that she'd get an accomodation whereby she did not attend lectures

[final cobalt]

I had her in another class and she was fine the rest of the time

But yeah, she might have considered that

[iron basalt]

While you are watching 3b1b, you can go over their essence of linear algebra and essence of calculus series.

[final cobalt]

If you're looking to learn Calculus

And maybe LA too, because I think he was working on it

https://www.youtube.com/watch?v=fYyARMqiaag&list=PLF797E961509B4EB5

This guy is the bomb. I learned all my Calc from him

[unkempt wigeon]

So if I'm getting this right from the current video that I'm learning on current knowledge vectors are just for graphing basically (X,Y)?

[final cobalt]

Vectors are a fundamental concept in many domains of mathematics

It its simplest, a vector is a value that has both a magnitude and a direction, and both of the qualities are essential to it's nature. From a linear algebra perspective, a vector is a list of numbers where each number represents some value along an axis of freedom - it is closely related to the notion of dimensionality

There are "linearly independent" aka "mutually exclusive" vectors, which mean that there's no combinations of vectors X and Y which can produce the vector Z, the same way there's no way you can move along the X axis or the Y axis in space to change your position along the Z axis

From a calculus perspective, vectors function a bit more like pointers. A vector says "from your current position, go in this direction for this long"

[wooden sail]

you're thinking "orthogonal" here, which is stronger than lin indep

[final cobalt]

True - I'm simplifying

[unkempt wigeon]

How can I make it so when there's a 3D array? Like for .stl and.obj front neural network to generate 3D objects after I figure out using a tutorial how to make a 1D array to the ETC

[delicate elk]

im trying to do a stochastic gradient descent, and having some trouble with calculating the linear regression derivatives, mean square loss derivatives, and the ridge obj derivatives, anyone familiar with stuff like that?

i cant distribute code, so i guess im looking mostly for guidance

[small wedge]

for the linear regression derivatives, what part are you struggling with?

[delicate elk]

im honestly not sure which are correct and which are incorrect, i think my mean square loss ones are wrong but im not sure how

after talking with some people their rmse for a specific sample is much different than what i got

[small wedge]

can you post your implementation of the MSE derivative? or your math that you used to calculate it?

[delicate elk]

-2*(y-x)/(y.shape[0])

but i got that from chatgpt so im not confident

[small wedge]

http://arxiv.org/pdf/1802.01528

so a mean is a sum / len, I see you dividing by the len but not applying a sum

need to see more of your implementation to comment though ig

why can't you post your code?

[delicate elk]

its for a course and in the instructions were not allowed to distribute anything

i knowi could, but i dont wanna risk it

[delicate elk]

np.mean(square_loss(x, y, th, th0), axis = 1, keepdims = True)
this is what we have for mean square loss

we have to find the derivative wrt th and th0

(y - lin_reg(x, th, th0))**2
square loss

np.dot(th.T, x) + th0
lin reg

[wooden sail]

th0 a scalar and th a vector? or the more general case with th0 a vector and th a matrix?

[delicate elk]

th0 is a scalar th is a vector iirc

[wooden sail]

that makes things simpler. how did you approach this? did you expand the square into a matrix-vector product?

[delicate elk]

honestly i dont really know what im doing here

any guidance is appreciated

[wooden sail]

.latex the standard approach is to note that [
(y - \bm{\theta}^T \bm{x} - \theta_0)^2 = (y - \bm{\theta}^T \bm{x} - \theta_0)(y - \bm{\theta}^T \bm{x} - \theta_0)^T,
]
since the transpose of a scalar is the same scalar. so multiplying those two scalars, we get the square we want for the squared loss. now you expand the product and again note that the transposes can be flipped judiciously (since the results are scalar) and find that
[
(y - \bm{\theta}^T \bm{x} - \theta_0)^2 = y^2 - 2y \bm{\theta}^T \bm{x} - 2y \theta_0 + 2 \theta_0 \bm{\theta}^T \bm{x} + \bm{\theta}^T \bm{xx}^T \bm{\theta} + \theta_0^2.
]
you can then use your standard matrix calculus here because you have scalars differentiated w.r.t. scalars, and scalars differentiated w.r.t. vectors (depending on whether you differentiate w.r.t. $\theta_0$ or $\bm{\theta}$).

[wooden sail]

@delicate elk so here differentiating w.r.t theta and theta_0 is a lot simpler

[strong ibex]

Hey
Does anyone here have idea about implementation of opencv or YOLO

[serene scaffold]

Anyone at the NVIDIA conference in DC? If so hmu

[river cape]

Any idea as to why it isnt reading the images?

[serene scaffold]

Did you confirm that either subdirectory contains files and that those files have the expected extension?

[river cape]

I did check them

They do have the files

I tried to do for a new notebook and then coded it again

the same result

[serene scaffold]

Can you expand these

[river cape]

@serene scaffold is it a problem with the path?

[serene scaffold]

Are you sure /content is right?

[river cape]

Yea how else would you access it

[serene scaffold]

I haven't used colab in a long time, so I don't know if that's the name of the user directory

I guess it is

[river cape]

yea it is

[serene scaffold]

Look at the docs for the flow from directory method

Maybe there's a caveat like all the files have to be in the directory root (not a subdirectory)

[river cape]

I did refer a video and it showed me the same way of exceuting it

[serene scaffold]

They might be using a different version.
Check what version you have and look at the docs for that version.

[river cape]

Over here , the folder architecture is explained but as per my code , it shoudl work right

[serene scaffold]

I can't help more at the moment.

[river cape]

Hmm its alright . thanks

[main fox]

Try without /content

and add a slash at the end of train and test

[unkempt wigeon]

May I ask a question

[rich moth]

Maybe its a permissions thing. Create some debugging to print out directory paths or verify the existence of the file with 'os'

[main fox]

I give you permission

[rich moth]

hold on, im not sure i concur

[spring field]

You have been in this server for a while, you know how this works, you just ask the question
Also stop apologizing for everything

[main fox]

Matiiss has given their wisdom

[rich moth]

i think you're on to something, have to see the root directory

[main fox]

If pwd gave "/content", they are inside content directory, so no need to specify the path as "/content"

Otherwise it's looking for /content/content/train

[spring field]

a leading slash at the start of the path is an absolute path starting from the current drive/root

[unkempt wigeon]

How can I specifically make a convolutional neural network because that seems like it would be the easiest to do I'm trying to make a neural network that can recognize any type of image live action or otherwise my apology

[rich moth]

remove /content/train try 'train' and 'test' it looks like you're in the content directory

am I crazy here?

[spring field]

^

[desert oar]

fast.ai and d2l have all the answers 😉

[main fox]

Yes, however the absolute path seems to not be working. What would you suggest to OP?

[unkempt wigeon]

How so?

[desert oar]

Because they are educational resources that teach you to do exactly what you have been asking about how to do for weeks

[main fox]

I see fast.ai mostly uses their own library
Would you recommend this library or does it eventually pivot to PyTorch for beginners?

[charred egret]

You should pick up a structured resource/tutorial such as the ones people here have been suggesting. Do you remember when you said you want to start on the hard parts? That will only work if you’re following good resources.

[charred egret]

Library doesn’t really matter in the grand scheme of things. If you’re learning the concepts properly and gaining the understanding of what it’s about. The concepts are transferrable. Ofc it depends on what your goals are. Do you want to learn pytorch (the library) or do you want to learn deep learning?

[main fox]

Once you learn the concepts you need production ready tools to apply them

e.g. how Tensorflow keeps track of shapes for you in CNNs

PyTorch is picky about the data types of your tensors, e.g. loss functions

Callbacks, early stopping, initializations, etc are all things you need to learn how to do in the library you choose

[charred egret]

Tools are usually the easy part. That’s why people say solve the problem before you even start coding

[main fox]

If you're gonna learn the concepts, might as well learn them using the right tools

Just seems inefficient to learn a concept separately from the tool you need to implement it in

[dry raft]

hey guys! 👋 i’m looking for some good image denoising techniques using neural networks. any cool methods or models you’ve come across? would love to hear your thoughts or any resources you recommend. thanks!

[main fox]

Checkout paperswithcode.com/task/image-denoising

[cloud relic]

hi, so I recently uninstalled anaconda on my macbook and now I can't even run python. Does anyone know why?

[serene scaffold]

Because you let go of something that was weighting you down to make way for something even better

You probably need to download and install python from python.org

[cloud relic]

i have python installed, but commands like "python" and "pip" dont work in my terminal

[serene scaffold]

What happens if you do python3 --version

Also I'm going to sleep

But I believe in you

Deleting conda was an amazing decision

[cloud relic]

thanks

[serene scaffold]

Things might be difficult right now. But soon you'll be tired of winning.

[cloud relic]

"zsh: command not found: python"

[serene scaffold]

I said python3

No space

It wasn't s typo

[cloud relic]

oh damn i got an output

3.12.6

[serene scaffold]

Show

Yay

[cloud relic]

yay

[serene scaffold]

You're winning again

Savor this moment

[cloud relic]

ok ill try to figure it out from here

you can go to sleep, thanks for the help

[serene scaffold]

Because soon you'll be tired of winning

And you'll remember this as the last time that winning felt good

[quartz lotus]

anyone know if the opencv annotating program is in the latest version of opencv?

[rich moth]

SuperAnnotate?

[quartz lotus]

this one

[twin relic]

Hi , please suggest me good youtube courses and resources to get started with Machine learning

[scarlet anchor]

Any of u here, good at Big data technologies like Kafka Hadoop?

[plucky condor]

Hi! I have a question regarding the TimesNet model, specifically the Time-Series-Library implementation of it (https://github.com/thuml/Time-Series-Library/blob/main/models/TimesNet.py). I was looking into the long term forecast and noticed the function took the following parameters self, x_enc, x_mark_enc, x_dec, x_mark_dec.

I found that x_enc represents the data from which a prediction is made, and that the x_mark_enc represents the time series features of this data (for example a timestamp)
(If I'm wrong about any of this please correct me)

My main question is about the x_dec and x_mark_dec. To me it looks like the x_dec represents the data that needs to be predicted (often respresented of y), and the x_mark_dec the time series features of this need to be predicted data. What I don't understand is that the forecast method does absolutely noting with x_dec and x_mark_dec. I understand that x_dec is not used since it is the thing you want to predict. However I would assume that x_mark_dec should be used since the model would just be trying to guess when the next data point is. So:
Why does the TimesNet model(or specifically this implementation) not use the x_mark_dec?

[spring field]

It seems to be an issue with the resource loader to an extent
Clearly it can find the path given, it just doesn't load anything from it (unless it silently fails or some stuff)

[river cape]

@rich moth @main fox I tried using non-augmented way of sending images in batches, turns out it reads those images

Guys I got my error

its class_mode = 'binary'

and classes would be the list of class folders like , classes = ['cats','dogs']\

[grand breach]

how can I increase recall score of my model for logistic regression trained on a dataset having high cardinality and high class imbalance, I've tried keeping few highly correlated features to prevent loss of any important information

are there any ways to tune my model ?

[vestal spruce]

Is anyone familiar with algo-trading? I just posted a question on #1035199133436354600, please check it out if anyone is willing
btw here's the post
https://discord.com/channels/267624335836053506/1293523804739473478

[vestal spruce]

I'm not familiar wit hhigh cardinality data, but for class imbalance you could resample the data, which could go two way, either oversample the minority class or undersampling the majority class, another method would be giving a class weighting on those class giving it more impact to the model, and remember to split the data not just randomly but also in ratio with the class ratio so that you have accurate representation of the data when training and testing.

[grand breach]

Yes i split using stratified k fold, i don't like doing oversampling because it puts a lot of artificial samples thus adding more noise

[vestal spruce]

then your option would be 2, which is undersampling or using class_weight then, also I'm curious if you're building the model using TF or SciKitLearn?

[grand breach]

using sklearn

[vestal spruce]

both does have class weighting so you might want to look into their documentation about it.

[jaunty helm]

sklearn's LogisticRegression (and many others) has a class_weight param you can set to 'balanced'

[grand breach]

yeah, i was reading about weighted LR this morning, should give it a try

ok that's increasing the recall to a decent value 0.62; earlier it was 0.01 but precision has decreased maybe because I've used refit=recall

[vestal spruce]

Hmm not sure, I don't really get the big picture of what you've made so far, might want to post your code as a thread on the #1035199133436354600 forum, so we can see and discuss about it?

If you have already made a post there just give me the link to it so I can start helping you out

[grand breach]

oh, ok let me try first if it persists I would create a post, thank you !

[vestal spruce]

Sure thing, just ping me if you need help with it.

[grand breach]

generally asking would using L2 or L1 regularization help here to tune LR ?

[vestal spruce]

Hmm since you're data is still underfitting rather than overfitting (recall and precision), I think using regularization can be excluded from the training and testing process for the time being.

[grand breach]

Makes sense

[vestal spruce]

IIRC regularization is used for overfitting

[grand breach]

Now i'm just thinking if my dataset really doesn't have a lot of linear patterns, maybe that's one reason why LR isn't performing or highly correlated features (multicolinearity) has just stagnated the performance

[vestal spruce]

If you don't mind sharing what dataset are you using for this LR model anyway?

[grand breach]

Yeah, it's on kaggle called Avazu CTR prediction

https://www.kaggle.com/c/avazu-ctr-prediction

[vestal spruce]

still looking into the dataset

[grand breach]

no problem it's quite huge

some 4000k rows

[vestal spruce]

Ok so from my understand and past experience, I suggest other model for this kind of dataset, since a lot of the features/columns usually have a non-linear relationships with the target/label, but also due to the imbalance nature of the dataset right (typically there are more non-clicks than clicks) So a better approach would be something like random forest, or whatever model used on a science article/journal about CTR, you might want to read about them first since half of the work is actually reading result from other people's works and experiment while also experimenting yourself, sometime you find new idea from it.

Like my final thesis was about Development of dialogue transcription of podcast audio using speaker diarization, and for the reference I read from Quan Wang's Speaker Diarization with LSTM.

and I gotten the idea to just combine pre-existing audio transcription model which was OpenAI's Whisper model with a clustering algorithm, and that works well

I'd love my paper but it's written on my native language, might want to translate it myself soon lol

[grand breach]

thanks for suggestion, one question how did you confirm that data has lot of non linear patterns ?

my approach was to run log reg and assess it's performance to confirm data is non linear

[vestal spruce]

To answer your question, is a list of rule set and logical understand already taught to me in my college days.

those rule sets where translated by my tutors as a deep understand of the dataset at hand, it was in his nature to fully grasp the nature of also every dataset he analize

Me personally still learning how to have his sense of intuition

[grand breach]

I saw some of research papers using MLPs or some special NNs, i'm using ML algorithms as i chose to do this as a ML project

[vestal spruce]

oh but for this case specifically a lot of the feature were categoricals so it was already in my check list that using different method like classification model would be best

[vestal spruce]

So you might want to try classification method that works with categorical data.

[jaunty helm]

that's pretty normal
better recall means your model got more of the total targets than before (i.e. if you had 100 fish, you went from catching 1 of them to 62 of them)
but that also means your model is a lot more lenient on what it might think is a target, thus precision falls (continuing from fish, it's like you're casting a wider net than before; more fish, but also more other things like pebbles or seaweed)

so it's usually a tradeoff, if you try to optimize recall, precision will likely drop as a result, and vice versa

if you want to improve one without the other falling, you'll have to come up with a better solution
i.e. use a more sophisticated model, good feature engineering, gathering more data, etc

[desert oar]

iirc it's built on pytorch, so i think you should be able to pivot on your own? but don't quote me on that

[desert oar]

what kinds of features? how many do you have?

you need to think practically about this. your model is trying to learn a relationship between your features and label. so if the model is performing poorly, you need to ask: is there actually a strong relationship here? if so, what is the nature of that relationship, and why isn't my model capturing it?

[glass pier]

is it reasonable to want to implement a gpt without automatic differentiation? i've only got a (trainable) embedding layer so far and differentiating that already took me a fair bit of figuring out (skill issues)

any advice for computing gradients of the other parts of the transformer? seems pretty daunting just looking at some 'blueprints' and how many parameters there are

[grand breach]

most of them are categorical even thogh other people have seperated some of them as numerical

[grand breach]

the question i'd in my mind was if i had not tuned it optimally to the fullest because (i know this is cheating) i saw other people getting decent metrics when using LR so i doubted myself and thought if i've not tuned it properly so i was stressing hard

[jaunty helm]

did you one hot encode the categorical variables? though you might get the curse of dimensionality
maybe use a tree-based model?

[grand breach]

oh my gosh, OHE would result in a really huge & sparse dataframe, i think already there are some 24 columns

i don't think it is a scalable option

[jaunty helm]

that would certainly be a problem
try a tree based model? those should handle categoricals natively
your LR might be doing worse due to how you're representing the categories (e.g. encoding dog=0, cat=1, bird=2 is not great)

[grand breach]

ok i used hash encoding and it is known to have collision problem

like two values might have same hash value

i read target encoding would be cheating as it has probablistic values

and might overfit on data

ok i'll try running decision tree or random forest to see and rule out if it is the encoding technique causing the problem

[jaunty helm]

why would 'having probabilistic values' be cheating?
there's also others like frequency encoding ig

[grand breach]

ok i was reading an article on medium that said that hash encoding is really a good technique

[vestal spruce]

Oh also If you'd like there are some cheat sheets for data science just to streamline the learning process

iirc a github user with complete cheat sheet for data science is abhat222, might want to check it out

[grand breach]

Exactly! i was going through cheatsheets for last couple of hours, will check out, thanks a lot !

also i used gridsearch to tune space dimension and found it was 64, i've categorical columns having 1000s of unique values

[grand breach]

rationale:

[scarlet anchor]

how do i upload a kaggle project to github 💀
is there some oss alt to n8n its only a 14 day free trial afaik

I want to automate workflow, not just copy paste code into pynb and push

[quaint rivet]

What loss function should i use in building detection? i am just doing basic level detection. I have tried binary cross entropy. It is not giving the desired result

[unkempt apex]

ahh, something interesting, you should provide more info

about your model first

and also what u wanna achieve

[fallow frost]

I have a bit of a challenge, I'm creating a script that does regular clean from a postgres db.
basically it moves all the rows that the PK dosent match (to get rid of the old ones).

I can use sqlalchemy which supports paramterized query with a tuple (WHERE pk IN :values), but then I'm loading all the data in memory, and I would need to dump it as a parquet or some other format.

I would love to use duckdb but they dont support parametrized query with a tuple (which is outrageous if you ask me), which would be great, cuz it would use basically no memory, anc I can use the COPY command to copy the results to a parquet direcly.

then I would probably save the parqeut on S3 like this: f'{table}/backup_timestamp={datetime.datetime.now()}.parquet'

what do you guys think?

[left tartan]

Uh, I've posted how to do this a few times. Ask me over in duckdb land and I'll link the post.

[fallow frost]

can you link a reply please

[left tartan]

done

[fallow frost]

thanks man!

[fallow coyote]

to the experienced persons here, how did you get into the ML/AI space and how did you begin to develop your skills in this space?

Even though I'm in uni, I wont acutally be getting into the ML stuff, or any coding in general, until next year (doing a foundation year; look it up if youre not famililar). Just feel like Im reaching a bottleneck again

[serene scaffold]

I switched majors from linguistics to computer science, and the computer science department's language technology specialist took me as one of her disciples.

[fallow coyote]

What was the learning process at the beginning when you first started in ML development (in terms of the prerequisite knowledge you had coming in, the beginning resources you used to get you started and the resources that brought your skills further)?

[serene scaffold]

It was a clusterfuck for the first several months

[fallow coyote]

I've been going through that clusterfuck for about a year now and I'm still not out yet. What resources though did you find the most useful (idgaf if theyre not beginner friendly, jsut want to understand your learning process)?

[narrow merlin]

its "something" hahaha, important is to really stick to the basic understanding, i think that helped me

like a lot of the things are just fuzzy details that you never will touch if you are not actual implementing something for real that you can "measure". But I think the biggest problem is still the propaganda and misunderstandings that are flowing around. I do an AI meetup on a freelancer platform and there was a guy 50 years< IT experience, 77 years old total crack, he loved the possibilities on ChatGPT and everything, and he really "understood" what he saw and he really realized the potential, but he never actually understood that he can literally run all that with a local model on his own hardware and he doesnt need a datacenter

and he used chatgpt for MONTHS

Luckily here on python codern this is less of a problem 😄 There we just have the langchain syndrome hehe

[dry raft]

why do people use StandardScaler for ML projects a lot? is it for easy standardization of data, or it is to enhance the data in some way?

[serene scaffold]

normalization. it doesn't "enhance" the data.

[dry raft]

ok, thanks very much! i used to see this a lot on kaggle when i was a beginner, now i get it, so thanks! this will be very useful for my reg-ression projects!

[fading wigeon]

Is there any sort of standard for determining convergence during training?

Like a change of less than 0.01% or something

[main fox]

You could set a threshold to meet for the metric you're tracking or set early stopping, where if you don't see improvements in your test loss for X amount of epochs, you stop and save the weights of the model with the lowest test loss

[serene scaffold]

When the rate of change for the loss starts to flatline

[fading wigeon]

Yeah, I'm just.... let me rephrase.

Theoretically I'm familiar with the concepts. I just don't know what thresholds to code in practice.

I did consider like... when the changes start oscilating about the zero point

[main fox]

Your test loss may not reach a zero point
You should instead see if any improvement happened over the previous epoch(s)

[fading wigeon]

Fair

[unkempt wigeon]

May I ask a question

[serene scaffold]

Remember to never ask to ask. No one will commit to answering a question before they know what it is.
But you should probably focus on following along with one of the many resources we've suggested you use over the past several weeks.

[unkempt wigeon]

Is there a theoretical limit to how many neurons can be home at work

[serene scaffold]

Be home at work?

[unkempt wigeon]

Sorry darn auto correct can there be a theoretical limit of how many neurons can be in a network?

[serene scaffold]

What do you think

And why?

[unkempt wigeon]

Well it would depend on the system and how many graphics units and RAM it has and helping the files are and how much time it needs to crunch so in theory there is nothing theoretical in it but there is depending on the system

[serene scaffold]

You are correct

There is no theoretical limit. Only practical ones imposed by hardware and our ability to wait for competitions to complete

[unkempt wigeon]

How many gpus would it take with the same amount of neurons in a human brain which is 86 million

I'm trying to judge this because I don't know if I might continue working on the same network but doing improvements like if I make a convolutional neural network then being able to have it also process audio and use it

[serene scaffold]

You're not at a stage where you can speculate about making a neural network that models human cognition

Neural networks are inspired by what was known about neurology at the time

But that's it. There's no guaranteed similarities

They don't necessarily "learn like a human does". That's just marketing.

[iron basalt]

Real neurons are something entirely different. For reference it takes about one convolutional neural network with about 5-8 hidden layers (IDR the exact specs.) to simulate the responses of a real neuron decently (to mimic it). This is actually an improvement over previous attempts using differential equations directly.

Also for reference, a single real neuron can solve XOR, can do complicated predictions on its own, and also there are many types of neurons.

(Also they don't really have a single weight vector, it's more like a set of weight vectors (it can do clustering (in a messy, very approximate, biological way)))

(The list keeps growing as we find out more)

[unkempt wigeon]

I know I'm wondering how big on your own network couldn't get in what's the ratio for gpus and RAM sticks to increase the capability of the network so it only takes a couple of minutes of training hours or years on a slow computer

[iron basalt]

The long training times is due to the way deep learning fundamentally works, it's not one/few-shot online learning, which is what biological systems do. Doing this as well as those biological systems would require a change in hardware architecture. There is some work being done on this, the largest (in terms of funding) efforts by Intel and IBM.

A single GPU already uses too much energy compared to a brain.

GPUs were designed for dense parallel linear algebra work (updating a lot of pixels on the screen).

Current algorithms (deep learning) are designed around this.

They also come more from a very math (statistics) background, which also affects the type of algorithms found. Since statistics is designed for stuff like science, where you collect a bunch of data upfront, and then run through all of it in post (offline).

Biology does not have time to collect a bunch of data upfront (nor a place to store it all and retrieve it super fast), you need to learn now to make decisions now, or not survive (the context is not science, but rather survival via stuff like reinforcement learning (agents)).

[iron basalt]

(This gives better, less biased results, but can only be done if you can afford it)

[unkempt wigeon]

I'm sorry if I'm prodding with these questions

[iron basalt]

(Humans have found (evolved) a hack around this limitation, by communication with others (speaking/language) (they store more data that you can retrieve), and writing (augmentation of human memory (gives permanent memory that can even go across generations well)))

[unkempt wigeon]

Is it possible to make a 3D convolutional neural network

[unkempt wigeon]

Since the human brain can stitch what to the brain is 2D into a 3D object is it possible for a neural network 10 do the same or even generate 3D files sorry

[iron basalt]

You probably are thinking of SLAM: https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping

Yes, there many artificial (and more biologically plausible) neural network based solutions.

[iron basalt]

Or you do mean generating 3D models (polygonal)? That is also a thing.

[unkempt wigeon]

I teach it how to generate 3D models of things with keywords so I can use it to generate 3D printing files based off of data if I need to quickly redesign a new robot I can just have the general Network sign one for any type of purpose multipurpose singular purpose I don't mind

[rich moth]

that sounds fun

[iron basalt]

Yes, that can be done and there are some projects / products for it.

[unkempt wigeon]

But right now I'm trying to do image recognition and I'm trying to use pillow library to grab the images from a training file for that into Data so I can do handwritten digits then go on to letters and then grammar then so on so forth

Does anyone know where I can download the handwritten digits where should I submit my own write them on regular paper with a pencil and then photocopy in and put them into Photoshop or whatever I can use make the image its own separate cell and then went through the network sorry

[quaint rivet]

Ok i am using unet model.

I am using a U-Net model to extract buildings from images. I understand that the model may not achieve perfect accuracy, but I aim for a detection rate of 60-80%. At the very least, I expect the generated masks to demonstrate some indication of the model's ability to identify buildings.

I have constructed a dataset using the Massachusetts building dataset. I am employing binary cross-entropy loss as my loss function. Currently, the generated masks are relatively small, as illustrated in the image.

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])


callbacks=[
    EarlyStopping(monitor='val_loss', patience
                  =5, restore_best_weights=True),
]


history = model.fit(train_xx, train_yy, validation_data=(val_xx, val_yy), epochs = 10, batch_size=10, callbacks=callbacks)

[final cobalt]

One thing worth noting is that (as I understand it) more neurons is usually the worse option

The gradients which arise from certain problems have certain natures in and of themselves, and they only need so many neurons to properly approximate the function

A better network trumps a bigger network is what I'm saying, I suppose. But don't take my word on that because I'm still learning myself

[unkempt wigeon]

Does anyone know where I can find the library for hand written digits?

[final cobalt]

MNIST?

Gimme a sec

[unkempt wigeon]

yes

[final cobalt]

import torch
from torchvision import datasets, transforms

# Define a transform to convert the images to tensors
transform = transforms.ToTensor()

# Download and load the training and test datasets
train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)

# Create data loaders for batching the data
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=64, shuffle=False)

# Checking the shape of the data
for images, labels in train_loader:
    print(f'Batch of images shape: {images.shape}')
    print(f'Batch of labels shape: {labels.shape}')
    break

Might be an error or two. I'm too tired to search for the code I wrote to do it, so I just had ChatGTP pull this up for me

But this should be the gist of it

The GNIST of it XD /pun

[unkempt wigeon]

#===[imports]===#
import matplotlib as mpl
from PIL import Image
import numpy as np
#================#

image = Image.open('empty')

array = np.array(image)

X = array

W = np.array([])

B = np.array([])



outputs = np.dot(X,W) + B

This is the way that I'm doing it I'm using just some pie in a few other imports one for graphing and one for getting the image turn into it already to be put in through the neurons

Is this ok?

[unkempt wigeon]

That was funny

[onyx frigate]

Hey guys so I finished python fundamentals all the way to oop and also did json what's the next step i need to take in order to create and fine tune llm using langchain and hugging face ?!

[agile cobalt]

hugging face is used almost exclusively for inference, not training nor fine tuning

idk if langchain has some fine-tuning support somewhere, but I don't think so, and even if it does, its focus is on creating pipelines that let connect LLMs to multiple forms of inputs and outputs (specially RAG, Tools, Agents - all high level concepts that are only used during inference, again, nothing related to training)

I recommend learning (in order):

• Numpy and PyTorch basics (working with arrays/tensors, indexing, broadcasting)
• Linear Regression, Loss & Gradient Descent
• how Neural Networks work
• how LLMs work (from how the input is encoded to which layers they use to how their output is sampled)
• how to fine tune Llama models

But you can just skip everything and throw "fine tune llama" or "fine tune gemma" in YouTube, the code is relatively simple if you ignore all the theory behind why it works and how to debug it if things work poorly

[onyx frigate]

So I actually want to create ai agents with langchain so they can access different tools also then I will fine tune the model for you know making it more efficient for the task I want to do with it.

[agile cobalt]

in practice are you'll never want to create a llm from scratch yourself though - training something like Llama requires millions of dollars worth of compute

you can create something comparable to GPT-2 with a reasonable budget, but anything beyond that gets pretty expensive

also fine tuning llms is not extremely common from what I've seen, now that you can just use prompt engineering instead (if you want to teach the model some information, use RAG instead - if you want some response format, use few-shot prompting with a few examples instead etc.)

[agile cobalt]

I would not worry about fine tuning until you have a working system in place

fine-tuning takes a bit of effort and kinda locks you to one specific model
prompting techniques can be applied to nearly any model, so you could easily swap from one provider to another or update to the newest SOTA model without having to re-fine-tune

[onyx frigate]

I think you're right making it from scratch is like refusing bricks and cement to build a house also RAG is a better option

[agile cobalt]

I think that the most common use case of fine-tuning right now is model distillation / generating a lot of example responses using a huge model, then training a smaller model on those responses to lower costs

(e.g. use Llama 3.1 70B or 405B to create 10000 example responses, then fine tune Llama 3.2 3B on those)

[onyx frigate]

Man this is like ai training ai 🤯

[lapis sequoia]

Hello there!! I've been wondering if there's a good entry point into AI & ML as a self-taught guy, I can't enroll in university courses so looking for just a widely accepted book that I could perhaps read!! (I am fairly good with Python imo)

[agile cobalt]

check the pins

[lapis sequoia]

tysm!! (somehow, it didn't ping?)

[peak thorn]

can anyone please tell me how can i load image dataset? it my first working with image dataset

[agile owl]

https://paste.pythondiscord.com/2RUQ

Can anyone help me figure out why these two implementations of what I intend to be the same architecture for an autoencoder have vastly different loss profiles on torch vs keras

[ionic temple]

Was generating ragas metrics for mistral and ran into
AttributeError: 'Mistral' object has no attribute 'set_run_config'
anyone has any suggestion or solution for the same. langchain_ollama doesnt work and I dont have enough credits for using the default OpenAI option. Have listed the issue here https://github.com/explodinggradients/ragas/issues/1466
Had to resolve this urgently.

[serene scaffold]

the code you showed does not contain set_run_config. please remember to always show the entire error message, starting from Traceback.

[ionic temple]

Sure my apologies for that

AttributeError                            Traceback (most recent call last)
<ipython-input-155-30d7c30fba2f> in <cell line: 34>()
     32 
     33 # Step 3: Run the evaluation
---> 34 results = evaluate(
     35     dataset=dataset,  # Use the Hugging Face Dataset object
     36     metrics=[answer_relevancy],

2 frames
/usr/local/lib/python3.10/dist-packages/ragas/_analytics.py in wrapper(*args, **kwargs)
    127     def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    128         track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129         result = func(*args, **kwargs)
    130         track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    131 

/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
    204 
    205         # init all the models
--> 206         metric.init(run_config)
    207 
    208     executor = Executor(

/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py in init(self, run_config)
    151                 f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run."  # noqa
    152             )
--> 153         self.llm.set_run_config(run_config)
    154 
    155 

AttributeError: 'Mistral' object has no attribute 'set_run_config'

this was the error message @serene scaffold

[unkempt apex]

looks good though

[unkempt wigeon]

what sould i use for a kernnal?

[main fox]

Assuming you're talking about CNN, and you're referring to size, stride and padding

[unkempt wigeon]

Yes I figured out what I might need for getting the size which would be the amount of input neurons I need to know how big to make the kernel so that I can get all the data to detect the curve and then add by a bias it's the output that goes into another layer until it gets to the final neuron

[main fox]

Well, there is no magic number. You have to try out different parameters and see what works for your task.

To get an idea of what parameters might work, you'll need to understand what happens to your input at each convolution (hint: the images get downsampled), and I'd recommend you look at popular CNN architectures. Check out the TinyVGG and see if you can replicate that. Assuming you're doing MNIST which are grayscale images, you'll also have to keep in mind you don't have RGB images, just grayscale. This means that your input is one "channel", not three.

[unkempt wigeon]

Do you know where I could find that library sorry

#===[imports]===#
import matplotlib as mpl
from PIL import Image
import numpy as np
#================#

image = Image.open('')

array = np.array(image)

X = array

W = np.array([])

B = np.array([])



outputs = np.dot(X,W) + B

what i have so far

[main fox]

https://poloclub.github.io/cnn-explainer/

[spring field]

have you made a simple feed-forward network from scratch in numpy yet?

also known as a fully-connected network
also known as a dense network
also known as an affine transform (network?)
also known as an MLP (hate that term)

[small wedge]

I hate the term ANN

as if we ever need the context that we're working with artificial nn's as opposed to natural ones

[spring field]

or rather, known as those layers not networks, but multiple of those layers make a network in the end (if you don't forget your non-linear layers too)

[storm valve]

I’m looking for an existing NLP corpus that focuses on Python-related vocabulary, including terms frequently used in Python programming. Currently, I’m extracting words directly from source code, such as imports, function names, and assignments, along with a small collection of common programming terms. However, I’d like to expand this corpus with more general Python-related terms to enhance its comprehensiveness. Any suggestions or resources for obtaining a richer Python-specific vocabulary corpus would be greatly appreciated. Thank you!
https://discuss.python.org/t/seeking-a-comprehensive-nlp-corpus-for-python-related-vocabulary/66515

[spring field]

does the glossary contain anything of interest to you?

[storm valve]

i tried my hand at parsing the glossary and incorporating it to my existing corpus but it was very messy, i'll have to try again soon

[spring field]

also collections.abc might be a neat source for terms

[unkempt wigeon]

#===[imports]===#
import matplotlib as mpl
from PIL import Image
import numpy as np
#================#

X0 = np.array([1,3,4,6.9])

W0 = np.array([9,4,3,0])

B0 = np.array([1,4,2,3])

output = np.dot(X0,W0) + B0

def sigmoid(X):
    return 1/(1 + np.exp(-X))

output1 = sigmoid(X0)

print(output1)

like this?

[serene wedge]

Hiii good day!

Is this for Machine learning?

[faint quail]

https://youtu.be/15d-3FqNH-g
Built my own Machine Learning library from scratch using cupy, numpy and tensorflow functions occasionally

[main fox]

What tutorial are you following?

[main fox]

Your X0 is an array that contains both integers and one float (6.9)
You called sigmoid on X0, not output

You're trying to do all this in numpy. Is your expectation to build a CNN in pure numpy?

[unkempt wigeon]

I'm trying to build it all in numpy without any tensorflow or anything else because if something happens to those apis because you never know what might happen I don't know if it got directly goes to the site plugs in the data that wants to be trained so might as well get comfy on using numpy because it's a universal basic for anything really in Python that you need a lot of mathematics for sorry

[main fox]

You can "freeze" whatever version of a package you use, so if there are breaking updates, you use whatever stable version you used to build your model.
Also, the package won't send data back anywhere. But even if it did, I doubt they'd need more data on how to train a CNN for MNIST.

[unkempt wigeon]

What about a new AI model type

[main fox]

These libraries are open source, you can see they don't send data anywhere.
Also, if you manage to build a CNN in numpy, you'll realize why people don't do deep learning in pure numpy. Back propagation would be terribly slow.

[unkempt wigeon]

Why would it be slow?

[main fox]

Several reasons
numpy doesn't have built in automatic differentiation (efficient computation of gradients), it cannot leverage GPUs like PyTorch and Tensorflow, it does not have a JIT compiler

[storm valve]

huh, how so>

[final cobalt]

https://drive.google.com/file/d/1ErhaAJu2qTF3IQWOUXkCWOIp-gkY1hpg/view?usp=drive_link

Not sure if this helps anyone

40000 mtg cards (20000 unique ones) with abilities sorted into activated, triggered, passive/automatic, and keyword. ChatGTP was used to parse and sort the cards

[faint quail]

we dont have access

[untold fable]

when i am over with maths

in machine learning

[regal light]

can anyone recommend a llm model for code optimization which gives response time in less than 10 to 20 seconds. Also it should be less in size

[jaunty helm]

that depends on your hardware? a model that runs in ~10 secs on a 4090 will probably take longer if you have a 4060 instead
and 'less in size' in comparison to what?

[regal light]

I'm mentioning about the download size of it. Regardless of the hardware is there any lightweight LLM which is used for coding related tasks

[jaunty helm]

again, that's not saying much; what filesize are you looking for specifically?

[final cobalt]

Damnit

Google Drive being a jerk again

[regal light]

10 to 20gb

[final cobalt]

https://huggingface.co/datasets/Skywalker27/Mtg-Cards-Unique-Art/tree/main

Now with pictures!

[jaunty helm]

then you're looking at a full unquantized 7-8b model, or an 8bit quantized 10-20b model, or a 4-bit quantized 20-40b model
maybe check out the Qwen2.5 series

[regal light]

okay tha nk you

[quaint rivet]

I didn't get

[onyx frigate]

Is there any compatibility issue with the latest version on pandas and numpy ?!

[jaunty helm]

not unlikely
a couple of months ago numpy 2.0 was released (that included breaking changes)

[wicked torrent]

Multi-Agent Reasoning Problem Solver library in Python!
I just published a Multi-Agent Reasoning Problem Solver library in Python!
Check it out here: https://github.com/hg0428/Mar-PS

All feedback, suggestions, and critiques are welcome.
If you build something cool with it, please show me.

[peak thorn]

can anyone please tell me how can i load image dataset? it my first working with image dataset

[serene scaffold]

There isn't one universal way to load datasets. Is there a particular library you're trying to use to do it?

[west phoenix]

I hope this is okay to ask here but I am taking a class in college for Data Aanalysis and finding it extremely difficult to follow along with my professor. Does anyone have any advice or practice suggestions to help me better understand the basics?

[grand breach]

has anyone ever used tomek links to undersample, it's been 50+ mins ever since i ran it and is still running

there are some 47k samples to undersample

is this normal behavior ?

[muted dock]

Anyone available to help me in a vocal chat to restructure my codebase into multiple packages but in a monorepo I have tons of questions

[thorny salmon]

Is it correct that there is no way to sample in a way that preserves exceedingly low groups and yets ensure relative balance at the end? Using Pands >= 2.0. I have a dataset of 7M records that I need to form subgroups/buckets that I need to evenly sample from. These are the specific categories that I already have applied in the dataset:

• medium_type (digital, traditional, fetch_all)
• content_rating (g, s, e, q)
• normalised_score (<0.2 is VLS, <0.4 is LS, <0.6 is MS <0.8 is HS else VHS)
• focus_category (ff, other and interest, 'interest' has strings of interest that I also want to make a best effort of sampling)
• color_bucket (19 different color types including 'full_color', combos for color dont apply to the "interest" focus bucket as that is very limit)

There should be even distribution at each level if I was to go in and analyse it. This means roughly 50% for each medium, 25% each for rating, 20% each for score_bucket, 33% for focus and 5.2% for each color_type.

This is for a aesthetic scorer to be used on a finetune that we plan to freely release. I dont want a particular art type to not be represented. Else we will fall into the trap of super contrasty images are highly rated but we cant rate 7M records. So I need to sample at most 70k records.

Spent about 4 days attempting different implementations to no avail

[unkempt wigeon]

But why does it not have a built_in automatic differention?

[jaunty helm]

dumb suggestion, but why not just df.groupby(['medium_type', 'content_rating', 'normalized_score', 'focus_category', 'color_bucket']).sample()

[thorny salmon]

https://tenor.com/view/batman-thinking-thinking-about-you-think-thinking-of-you-gif-20503421

So my thinking was focus_category has a particular type (called interest) that is quite low in population but was important I sample @jaunty helm again to avoid the 'it can only rate contrasty pics and pics with feminine traits'

Some examples of this is, it contains vehicles, landscapes, cityscapes, mechas, concept art etc.

I want this scorer focus on composition and the quality of the work, not the contents of it. This I assume means relatively even distribution of the attributes above

[jaunty helm]

then what's wrong with the groupby().sample() method above?
if you do .sample(10) for example, you should get 10 samples for each unique combination of the 5 columns

[thorny salmon]

Sec, going to rerun it and spit out the results to sanity check

I recall getting poor results doing this

Running now.

[main fox]

It wasn't built for deep learning

[thorny salmon]

Well forcing me to sanity check made me compromise and make some of my groups a bit more general, results seem promising but waiting to see result of 60k output instead of 10k.

(I really wanted to ensure I had some of every color_type but ... I think I am going to send myself mad)

Also, how did the bot know that was a batman gif and react to that?

[jaunty helm]

is the data processing taking long or

[thorny salmon]

No me making decisions did.

I was uhmming and ahhing about whether to compromise on the color type bucket, took it from 19 to 4

[jaunty helm]

if your message contains bat in anyway sir lancebot's gonna do that im pretty sure

[jaunty helm]

does your focus_category actually only have 2 types other and interest, or does it actually store other and Vehicle and Landscape, etc

[grand breach]

ok it makes sense now, tomek link is a computationally expensive algorithm with O(n^2) complexity that calculated euclidean distance for every sample of n samples... waste of time for me, i think my data has very few majority class samples closer to minority class samples as I could see no difference... my dissappointment is immeasurable

[thorny salmon]

Just 3 types: ff, male/other and interest

[jaunty helm]

hm
but still, it could be that there's just no interesting type with the color bucket 7 for example

[thorny salmon]

Yeah... its tricky. I wanted these underepresented buckets with things like vehicle and landscape to be rated by us too

[jaunty helm]

don't think you can do much about that other than get more data lol
ig you can try oversampling? (the few times I tried working with them didn't work out so well tho)

[thorny salmon]

70k records is about the limit of what we can humanly do here unfortunately. We are elo rating them with glicko2 + pre-seeding their starting elo

Which means at best 15 battles per record

and that means...

70,000 * 15 (clicks) * 5 (seconds to make a judgement) = a month of work in hours 💀

[unkempt apex]

the all functions which you are using are right!
so just check your epochs or other parameters

[pulsar crow]

Hello

[quaint rivet]

issue fixed

[unkempt apex]

how?

[thorny salmon]

Thank you btw

Oh, does python discord not have points? If it does how do assign a “thank you, you helped solve it”

[serene scaffold]

we don't--we don't want to gamify the system

[thorny salmon]

Icic

[unkempt wigeon]

Is it possible to make it speed up?

[serene scaffold]

No.

Just use pytorch.

Or you can use JAX.

[unkempt wigeon]

I never went to academia for such knowledge and you can only access pi torch if you have a certificate in a field as far as I'm aware

[serene scaffold]

That's just entirely false.

It's free and open source software.

[worldly dawn]

Just for completeness, check out https://pytorch.org/ and follow the big button "Get Started"

[urban canopy]

Anyone know of open source AI initiatives?

Where the training data is also open source.

[worldly dawn]

AI is a field, so it's orthogonal to being OSS

do you mean a LLM?

[urban canopy]

Yes. Blender is to Maya as ??? Is to ChatGPT

[worldly dawn]

Right. So you are asking for OSS alternatives of ChatGPT, not OSS alternatives of AI

[urban canopy]

I am also interested in AI art but let's focus on GPT for now.

[worldly dawn]

on my todo list to dive deeper, but https://www.together.ai/blog/redpajama might be of interest

[main fox]

Neither did I, you don't need a degree to learn these things. You've asked several questions here and many people have linked you to great resources to get started. You should follow the advice given and try to go through one of them.

[lapis sequoia]

Hello! Everyone.. I'm a Bachelor Of Sciences in Data Sciences, I just joined the python discord server after a half month. I recently applied for BS Data Science and hopefully to learn more in this field with your help and with my own learning.

[final cobalt]

Welcome!

I hope you enjoy assaulting your own brain with knowledge humans weren't meant, biologically speaking, to comprehend on a regular basis

As well as torturing yourself with meticulous dataset collection and annotation, and the debugging of ephemeral and ill defined gradients in systems that themselves are also ill defined XD

[fallow coyote]

Word of advice: there is no shame starting at the basics. Remove your ego and listen to those who have more knowledge and experience. Thats what I do. Your foundations must be strong before you build anything on top or else, everything will collapse

[left tartan]

I had other plans for today, but: https://youtu.be/rbu7Zu5X1zI?feature=shared

[spring field]

well, there's cupy
but at that point, what are you even doing not going a step further with a lib that has auto diff as well...

[serene scaffold]

The use case for cupy is even more limited now that there's JAX

[spring field]

yep, but it would be an almost numpy equivalent, but faster 😁

[valid otter]

Budget laptops for AI/ML (less than $1000)

[serene scaffold]

Don't buy an ML laptop. You'll be overpaying for the amount of compute power you get, and it also won't be enough.

Just get a conventional laptop and rent cloud compute as needed.

[lilac saddle]

Guess all AI/ ML laptops aren't worth the price. Cause all the features work semi well

[merry ridge]

I am surprised that this is even a product. Does it just have a marginally better CPU and more VRAM than a gaming laptop?

[serene scaffold]

I've never heard of a laptop being marketed for AI

Gaming laptops are already pretty bulky and a worse value for compute ability than desktops

[lapis sequoia]

Your reply sounds like a bot, also thanks for encouragement.

[unkempt apex]

any specific ways to run .pth files ( trained model files ) on 512 MB ram?

the model accepts image ( grayscale ) and returns transformed image ( RGB )

model size is 6M param

which is nearly 150 mb

[final cobalt]

XD

Sorry. I was feeling bit a peaky last night, I suppose

[ionic temple]

Hey guys urgently needed a way around or a fix for this any suggestions or solutions will be highly appreiciated - https://github.com/explodinggradients/ragas/issues/1478#issuecomment-2407928155

SideNote - Have to go with ChatOllama as I dont have enough credits for using ChatOpenAI

[lapis sequoia]

Hey, In university we started learning programming fundamentals with C++.
But I was excited that they will teach us programming fundamentals with python programming.
What is your opinion?
Is it correct to start with C++.

[agile cobalt]

why are you using __setattr__?

[agile cobalt]

try asking in their discord server, they link one in the github readme

[ionic temple]

Tried it but got no replies from there side:(

[ionic temple]

Additonal check on default removal of openai

[agile cobalt]

use x.y = ... normally, you should pretty much never call dunder methods directly

[final cobalt]

In my opinion, the best approach is to start with Python and use it to get comfortable with the basic of programming - functions, objects, procedural thinking

But don't get too comfortable. Once you start settling in, switch to C(++). In my personal taste, C is less useful than C++. A good C++ compiler tends to write C code that's at least as optimized as human written C code, and it has features like classes and exceptions. Others might have other opinions

You'll want these languages because they are, generally speaking, the foundation of all the other languages you'll probably be using. In the least, they encompass the core concepts. You'll also need to be able to write fast code from time to time.

Once you've got that down, specialize as you need. Certain languages are better for certain tasks. Personally, I've developed a taste for Cython

It's a very happy medium between C(++) and Python. You'll probably also want to learn Javascript - but beware: Javascript is a friendly, well documented, universal dumpster fire of a language

Also, I (personally) don't think we'll be hand coding websites much longer. Web development is mostly a solved problem, and there are some very robust WYSIWYG tools like Webflow which cut the time to small/medium site development by 90%

[lapis sequoia]

Yeah, I Know that. I surface touched JavaScript while learning web development.

[final cobalt]

I love JS

I do, I thinks its...

Adorable

[iron basalt]

It does not really matter which language you start with, programming is a skill that is not tied to a language. Whichever you start with, I recommend at some point learning at least 2 entirely different languages (e.g. Python -> C++ -> Haskell).

(Also at some point learning how these languages can interoperate (try making some Python bindings for a C library that you made at some point))

[iron basalt]

Whichever language makes you want to program more is probably the best starting language for you.

[muted prairie]

how can i make my python run green like this

[tidal bough]

A good C++ compiler tends to write C code that's
(C++ compilers generally don't compile to C at any stage of the compilation process. (I went down a small rabbit hole making sure this is true because I found out that clang used to be able to do that, but LLVM removed the feature allowing the translation of LLVM IR to C back in 2012).)

[final cobalt]

😮

I didn't know that

[muted prairie]

😡

[unkempt apex]

GREEN = "\033[92m"
this is green ANSI code
print(f"{GREEN}hello")
and this is how u can use it

[stoic hollow]

Thoughts on the Gemini api vs chatgpt one? Considering trying geminis free tier but not sure how reliable it is in comparison to chatgpt

[left tartan]

The short answer is that 'reliability' isn't a good measure here. They're all unreliable, and their utility depends on what you're trying to do and your expectations.

[stoic hollow]

Looking at integrating it with a job application project someone recommended me after having no luck since may with over 800 applications sent off

So basically wanting to know if it’ll actually work or if it’s going to run into issues

I usually run ollama for other projects but cloud is becoming more convenient atm but havnt played around with either or there apis

So basically trying to workout which will show most consistent results or if both are feasible

[final cobalt]

I call my ChatGTP Winston

[stoic hollow]

Nice you tried the Gemini one? Only considering it cause has a free tier

[final cobalt]

Nah

I pay my $20 a month for Winston

And I'm happy

It's a reasonable price considering how much use I get out of it

[stoic hollow]

Oh you don’t use the api? Or do they have a fixed price?

[final cobalt]

Oh! The API

[stoic hollow]

I wasn’t sure how often api calls are referred to as a request

Yeah I don’t mind paying a fixed price but was iffy about pricing

[final cobalt]

I mostly use the interactive version. It's a great teacher. I've used the API as well

The interactive version is fixed. The API is by token. They stretch pretty far, but it depends how much work you need done and how complex the task is

[stoic hollow]

Like I’ve used aws stuff as well but just like to ask around before I throw myself into something that has per use pricing

[final cobalt]

I had it parse the text of 20000 magic cards for about $20 of api credits

I thought that was very reasonable

[stoic hollow]

Ohhhh that’s not to bad at all so what does it count as a token tho?

Is it a request or per word

[final cobalt]

I think it's 4 bytes

Something like that

[stoic hollow]

Ah gotcha thanks appreciate it

I’ll try Gemini first then since it has a free tier and gpt if that doesn’t work out

[jaunty helm]

depends on the model
https://huggingface.co/spaces/Xenova/the-tokenizer-playground

The Gemma series is also by Google so maybe it's an OK estimate

[stoic hollow]

Think I need to host on ollama for Gemma tho right?

[jaunty helm]

or through some other service like OpenRouter
I was trying to say that what a token is differs from model to model, and using Gemma might be a decent estimate of how Gemini tokenizes things

[stoic hollow]

makes sense thanks

[untold fable]

Any stanford or mit or harvard student here

[serene scaffold]

What question would you ask them

[untold fable]

how to get there

[serene scaffold]

I have cousins who teach at Stanford and MIT. I'll let you know if they have more specific advice than "have good grades and do a lot of impressive things"

But I suspect that they don't.

Both universities get a lot of applications. There's a point at which it's a crap shoot.

[shut yoke]

Get incredibly good grades, be talented at something, do extracurricular activities and contribute to your community somehow. Besides the tuition fees you pay them, they need to benefit off of you

Your application has to be as perfect as it can get, so let's not forget an outstanding essay

Something original, something that stands out

Be different from everyone else, oh and use your victim card. I wouldn't mind lying if that would make me look better

Oh and don't forget the 💰

Don't expect to study there if you can't afford it

Maybe you'll manage to get a scholarship but it's still not cheap

[vale parcel]

I notice a lot of people struggle to get into AI, especially RL, so I created a simple GUI for making your own RL agents in seconds. I'd love to hear feedback from you guys 🙂

https://github.com/DQN-Labs/DQNSuite.git

[small wedge]

that's a fun project, good job

[stone patrol]

Hi, any AI dev who can help me to navigate threw process of becoming an AI dev using python

[worldly dawn]

In terms of career, a degree will be the path of least resistance and with the most opportunities and compensation

[stone patrol]

if i understood you correctly: if i will go to university and finish it, after i will get more opportunitites than without a degree?

[worldly dawn]

indeed

[stone patrol]

Answer dm pls, i want to know more

[worldly dawn]

I don't do DMs, just ask here

[frigid jewel]

micropython AI

I can manually overwrite it tuple

[stone patrol]

Which degree do i need?

[stone patrol]

And are you an AI dev??

[ionic temple]

Anyone on this please!

Hey guys urgently needed a way around or a fix for this any suggestions or solutions will be highly appreiciated - https://github.com/explodinggradients/ragas/issues/1478#issuecomment-2407928155

SideNote - Have to go with ChatOllama as I dont have enough credits for using ChatOpenAI

[left tartan]

Sorry to pass you around to different channels, but this is why it's best to ask your question directly. Sounds like you want to know about AI as a career. #career-advice is the best place to ask that. If you want to discuss AI concepts, this channel is good.

[muted prairie]

thank you bro

[vale parcel]

Thanks!

[noble axle]

guys I did lasso,ridge, and linear regression on the same dataset and the results (mse, mae, r^2) are all essentially the same. what in my data could cause this?

[quaint mulch]

do they also make the exact same predictions?

maybe the regularisation coefficient/strength is too weak?

is the error zero? the problem is solved?

maybe there is just no linear correlations in the data in the first place?

[noble axle]

no linera correlation would cuase this? i do have linear correlations between some features and the target

chat gpt said the opposite they said if a model does have a lot of linear correlation it will mean that lasso and ridge will perform basicalyl the same as linear

[charred egret]

How big is the dataset?

[noble axle]

13 columns 11k rows

is that too small to do regressions?

[quaint mulch]

not too small imo

[quaint mulch]

have you manually check the 14 coefficients from the 3 differnet models?

[noble axle]

yeah theyer ebasically the same give or take 1 or 2

[quaint mulch]

any there anything suspicous about the those coefficeints?
like some are super big, or some are very close to zero or one?

[quaint mulch]

btw, I just realized, this is the same like what I said. I said that it could he that the problem i solved, the errors are closed to zero

[noble axle]

i dont think so

[quaint mulch]

this is test set performance?

[noble axle]

yeah what do u think about it

is that good

[quaint mulch]

I think it make sense to me.
It seems that feature 1 and 2 (counting from zero) are the most important features that predominate everything else. The coeffcients are wayyyy bigger than everything else.
I'm surprised that ridge is performing very similarly to everything else, given that the coefficient are very different.

Did you use standard scaling and PCA?

[noble axle]

ok thx let me ask you one more thing which of the 3 regressions is most useful or most used in the industry

[quaint mulch]

idk, i'm not in industry lol

[quaint mulch]

I think what's useful is doing 3 things like you did, and try to understand what's going on, like what you are doing

[quaint mulch]

for instance, I guess there's no distribution shifts between train and test, so I don't think the models overfit on train and that .82 r^2 is a pretty reliable number i guess

[twin relic]

Hi, what are the prerequisites for the book Hands on machine learning with sckit learn , keras and tensorflow ?

[rough sigil]

Hey, I’m kinda new to python and I learned the basics but I’d like to get into data science. I know it’s unlikely that anyone here has the time for this but if anyone does, I’d really appreciate the help if you can get me started on it. I haven’t been able to understand the things I saw online but I think having someone show me what I need and what to do would help. Thank you!

[left tartan]

Don't worry about data science right now, just focus on learning Python and getting good at it. Ask for help/guidance in #python-discussion . It takes a fair amount of work to get through the first phase of learning programming, and worrying about specialization now is too early.

[left tartan]

Once you're through the beginning, you can look at resources like kaggle.com/learn to learn some data-related skills.

Separately, there's many Data Science topics you could learn... from various math topics to theoretical concepts to applied.

[unkempt apex]

sick man!

any plans to add progress of training also??

like showing loss functions and average reward throughout the episodes?

I would love to do that, because when I made my Pong RL game I have struggled a lot for this stuff

[shut yoke]

Computer Science or Data Science

[feral smelt]

Btw anyone wanna be study buddy can dm me...i like maths mainly linear algebra probability statistics and ai

[shut yoke]

which year u in

[feral smelt]

I m in my 4th year

I m a mechanical engineer graduate as of now but I will shift

[shut yoke]

dude Im 1st year 😭

[shut yoke]

computer science

[feral smelt]

No ds and ai

Aiming for iit or iisc

[shut yoke]

yeah but those are in optional courses I can take on the 3rd and 4th years

AI

[feral smelt]

Good

Artificial intelligence

Were are u from?

[shut yoke]

Morocco I study in Canada

[feral smelt]

Ohk

[rough sigil]

Is that a good place to get started on it? I think I know just about everything I need to branch off into something specific (game dev, web dev, data science, etc, but just python)

[left tartan]

That's a good place to get started on the coding part of the data journey. Theres plenty of other places for the theory part.

[rough sigil]

Wdym

[left tartan]

Data Science has a lot of science. Theory. Concepts. Math. Etc.

That's separate from learning how to do data stuff with Python

[rough sigil]

Could you explain some of it in dms?

Or is that a bit much

[left tartan]

Not really, but check out this channel: https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&feature=shared

[quaint mulch]

I haven’t been able to understand the things I saw online
Start with something something specific you saw online, and tell us what you understood and what you don't.

[quaint mulch]

also, pinned messages

[quartz lotus]

can someone tell me how to make positive samples in open cv. I'm following along the docs and it says to use the opecv_createsamples application but I don't see it in the open cv download

[unkempt wigeon]

Do I have to use tensorflow if I was going to do a machine learning I have everything for plotting the data taking the image and then converting the image into an array to be able to have it learn sorry

[unkempt apex]

ahh, there is a cheatsheet for this

https://images.app.goo.gl/AZ8nkqBjzPxuK7em9

you wanna remember how this color code works?

[small wedge]

you mean like computer vision?

[unkempt apex]

ohhh shit you want to detect it

[small wedge]

4 color masks

[unkempt apex]

lol

[small wedge]

get the average position of the white after masking for each

that tells you where each color is in relation to the others

[unkempt apex]

all images have green bg?

[small wedge]

where are these pictures coming from, can you standardize what they will look like?

[unkempt apex]

you got dataset or what? for this all?

share

if it's on kaggle

https://www.kaggle.com/datasets/eralpozcan/resistor-dataset?select=180K_1-2W

I only know this one

bruh what?? u deleting all msg?

[small wedge]

lmao

bro went scorched earth

[unkempt apex]

lol

[remote stream]

anyone knows data annotation software like ai based if its paid its fine

[river cape]

Hey guys , is it possible to build an ai model which reads the 2d floor plan and gives a 3d model of the building?

[vale parcel]

Thanks! Glad you like it. I did have plans of adding matplotlib and showing live stats as the training happens. It's currently not being worked on, but if you would like to contribute to the repo by making that feature (or any other feature you would liek being added), feel free! I'll take looks at PRs as soon as possible.

[untold fable]

A very big accident happened today 5 PPL died in a car accident and four are computer science student of my college 1st year,4th year and two were 3rd year

[quaint mulch]

you don't have to use tensorflow, you can use other things

[quaint mulch]

depending on your use, you can try using something like chatGPT

[quaint mulch]

Is it possible? Yes. Will it be good? That's a completely different question.

[scarlet anchor]

is there any good LLM for answering IoT related questions?

[quaint mulch]

What do you mean by IoT related questions?

[unkempt wigeon]

What other options my apologies

[scarlet anchor]

something related to embedded and internet of things

[scarlet anchor]

@quaint mulch

[river cape]

How do you start?

[uncut plaza]

Hey everyone, I have a problem and was wondering if there's an algorithm or machine learning model that can extract specific information from a bunch of text files. I have several resumes saved as separate .txt files, and I want to automatically pull out details like name, phone number, education, and other relevant information into an Excel or CSV file. Since each resume has a different format, I can't do this manually or with Excel, so I'm looking to use machine learning for the task.

[serene scaffold]

this is actually a really common problem. if you just look up "resume parsing with AI", you'll get a lot of options.

[untold fable]

It hurts when your batch mates lost there life's in a car accident

[tulip wyvern]

Does anyone know why I'm getting different accuracies when it should be equal?

rf_model = RandomForestClassifier(
    n_estimators=50,
    min_samples_split=10,
    min_samples_leaf=1,
    max_leaf_nodes=None,
    max_features='sqrt',
    max_depth=None,
    bootstrap=True,
    random_state=42
)

rf_model.fit(X_train, y_train)

rf_probs = rf_model.predict_proba(X_test)
y_pred = rf_model.predict(X_test)

correct = 0
for i in range(len(y_test)):
    if y_pred[i] == y_test[i]:
        correct += 1

correct_prob = 0
for i in range(len(y_test)):
    if np.argmax(rf_probs[i]) == y_test[i]:
        correct_prob += 1

print(correct / len(y_test)) => 0.48223401060954857
print(correct_prob / len(y_test)) => 0.007606846161545391```

because shouldnt np.argmax(rf_probs[i]) be the same as y_pred[i]

[tidal bough]

maybe rf_model.classes_ is in the wrong order compared to the dataset

[tulip wyvern]

oh jeez do you know how i would fix that

because my y_encoded array is just integers from 0-365 inclusive and my rf_model.classes is also just integers 0-365 inclusive

i dont think the rf_model.classes ordering is just a translation of y_encoded array because I ran this:

for n in range(-365, 366):
    for i in range(len(y_test)):
        if np.argmax(rf_probs[i]) + n == y_test[i]:
            correct_prob += 1
    prob = correct_prob / len(y_test)
    if prob > 0.3:
        print(correct_prob / len(y_test)) => never printed
        print(n)
    correct_prob = 0

print(correct / len(y_test)) => 0.4839723041415566

and the prob was never over 0.3 so i think the order is messed up in some crazy way

Edit:
NEVERMIND I GOT IT!!

[final cobalt]

Anyone here ever have any real luck training LoRA?

I understand the theory and most of the mechanics in theory

But no matter what I do, I can't get a decent result

[unkempt wigeon]

What else could I use?

[serene scaffold]

pytorch or JAX.

[fresh bay]

is there something wrong with this training call when I am looking at my gradients it really doesnt look like anything is flowing back?

for epoch in range(num_epoch+1):
    criterion = torch.nn.CrossEntropyLoss(reduction='none', weight = class_weight)

    for m in model_dict:
        model_dict[m].train()
    
    num_view = 1

    optim_dict["C{:}".format(i+1)].zero_grad()

    ci_loss = 0

    ci = model_dict["C{:}".format(i+1)](model_dict["E{:}".format(i+1)](data_tr_list[i],adj_tr_list[i]))    
    
    c1_l0_norm = np.linalg.norm(model_dict["C1"].clf[0].weight.clone().detach().numpy().flatten(), 0)
    
    gc1_l0_norm = np.linalg.norm(model_dict["E1"].gc1.weight.clone().detach().numpy().flatten(), 0)
    
    gc2_l0_norm = np.linalg.norm(model_dict["E1"].gc2.weight.clone().detach().numpy().flatten(), 0)
    
    regularization_term = gc2_l0_norm + gc1_l0_norm + c1_l0_norm
    
    ci_loss = torch.mean(criterion(ci, labels_tr_tensor.squeeze())) + reg_penalty * regularization_term  

    ci_loss.backward()

    optim_dict["C{:}".format(i+1)].step()

    loss_dict["C{:}".format(i+1)] = ci_loss.detach().cpu().numpy().item()

That change just looks to small in the loss given how large it is tbh

[unkempt wigeon]

What are the differences between the two because I want to get the best use across the board

[serene scaffold]

Just use pytorch

[lapis sequoia]

Hello! Everyone.

What does a data scientist do?

Is it worth it, or not?

[serene scaffold]

There's no rule that companies have to follow about which of their employees get to be called "data scientists". And the diversity of job responsibilities reflects that.

If you like statistics, then it's as good a job as any.

[lapis sequoia]

I think I made a bad decision by taking the Data Science Program.

[serene scaffold]

Is this at a university or what

[lapis sequoia]

University.

[left tartan]

Many Uni students change majors, at least in the US.

[lapis sequoia]

It's easy to change majors/program but the thing is that our parents will think that our child don't want to study.

The big problem.

[faint quail]

is there anything wrong with this backpropagation through a Concatenation layer? I'm passing the "node_values" (gradient w.r.t the output) to the layer before it by adding the two "node_values" together before passing them, but I'm concerned this isn't correct. Please help I'm self taught

        return _node_values + residual_node_values, [gradients, residual_gradients]

https://paste.pythondiscord.com/EV7A

[umbral lotus]

oh why say so?

I'm considering pursuing a master's degree of DS after undergrad

[rich moth]

Who is going to school you or your parents?

[quaint mulch]

That really depends on what kind of questions are you asking.

[quaint mulch]

https://gymat.github.io/SurfelNeRF-web/

and then you can combine it using something like marching cube: https://www.matthewtancik.com/nerf

https://bakedsdf.github.io/

[scarlet anchor]

Umm , all i want is a LLM 💀

[agile cobalt]

I wouldn't consider any LLM good at all, but you can try chatgpt or llama, just remember that you must double check pretty much everything any llm outputs

[quaint mulch]

Like, if you want to connect a timeseries data stream from IoT sensor to an LLM and you can ask question about it interactively, maybe something like NextGPT can do that, but only if that IoT happens to be IMU or audio, and not if it is like, ECG. In that case, you might need start your own research project.

If you have some text questions, and you want some text answers, like "what does iot stands for", then yea, use chatgpt.

like idk what you want

[sour parrot]

Could you please help me for this question

https://stackoverflow.com/questions/79090640/how-can-i-segment-the-handwritten-lines-in-this-type-of-documents

[quaint mulch]

I'm going to assume that's Arabic, so maybe find some arabic ocr?

[serene scaffold]

They only want bounding boxes around each line. They're not trying to transcribe it.

[dusky abyss]

looking for a mask segmentation model which I can use to automatically select background, head, body of a human etc given an portrait image

prompt based SAM has issues using the prompt, if I say background it will select the entire image, if I say body below neck it will ignore parts of the body like the hands, shirt below the suit etc

it isnt generalizing well

[tacit plinth]

Hello
Can anyone know to how preprocess NxN excel file to generate text before embedding and vectorization for LLM?

[mint plume]

Hello everyone! I've been in this Discord for a long time but I'm going to try to be more active here.

[thorny geode]

hi, im a high school students trying to self learn statistics and programming, is there any projects that is suitable for a high school

[mint plume]

For the record, I recent graduated from uni and I'm used to doing everything in R.

[broken eagle]

https://www.youtube.com/watch?v=AzRz6CEizJ4

Anyone familiar with replicating these kind of audio source separation models?

[river cape]

Hi guys how long deos it take to train a cnn model?

[charred egret]

It depends on what you’re doing. Too many factors to consider

[river cape]

model = Sequential()

model.add(InputLayer(input_shape=(224,224,3)))
model.add(Conv2D(32,kernel_size=(3,3),padding='same',activation='relu'))
model.add(Conv2D(32,kernel_size=(3,3),padding='same',activation='relu'))
model.add(MaxPooling2D(pool_size = (2,2),padding='same'))

model.add(Conv2D(64,kernel_size=(3,3),padding='same',activation='relu'))
model.add(Conv2D(64,kernel_size=(3,3),padding='same',activation='relu'))
model.add(MaxPooling2D(pool_size = (2,2),padding='same'))

model.add(Conv2D(128,kernel_size=(3,3),padding='same',activation='relu'))
model.add(MaxPooling2D(pool_size = (2,2),padding='same'))

model.add(Flatten())

model.add(Dense(512,activation='relu'))
model.add(Dense(256,activation='relu'))
model.add(Dense(200,activation='softmax'))

I am trying to do a bird classification model

[charred egret]

Still doesn’t tell you anything. Depends on the hardware, library you’re using, hyper parameters, what you consider as “done training”, and many more. Best way to find out is to just run it

[river cape]

Like I want to know the amount of time

I am using google colab

t4 gpu

[charred egret]

The only way to know is to run it

[river cape]

OH okay is it normal to have accuracy of 0.0074 in the intital epochs?

[charred egret]

you can guesstimate, run it in X number of epochs and time that and you can find out the time taken for 1 epoch. It’s going to be kinda close

[river cape]

Got it