#data-science-and-ml

1 messages · Page 122 of 1

wooden sail
#

all estimates are random variables and have their own statistical distributions

past meteor
#

Yeah, the MSE and the variance are the same under the condition they estimator is unbiased

wooden sail
#

specifically, the variance of the error and the MSE of the error

past meteor
#

Idk if you'll ever do reinforcement learning @spring field but there understanding and working with the bias of estimators has practical tradeoffs

#

Whereas in stats you can argue it's a bit fuzzy, unless you're making new methods and you want them to have certain sane properties

wooden sail
#

i'd also suggest to take some time to digest the idea. i've found a lot of people struggle with it

past meteor
#

Honestly depends on what you want to do though

#

It's something I had to learn in statistics and it goes into the bucket of things I never use

spring field
wooden sail
#

i would say you do use it. this is one of the nails in the coffin of trying to say whether your model is good by looking at the values of the MSE 😛 a mistake many people make

past meteor
#

Maybe having passive knowledge of it helped me? We can never answer that question

past meteor
wooden sail
#

certainly

#

we did recently submit a paper exactly on this matter btw

#

since it turns out a lot of results are misinterpreted in papers

#

the MSE do be tricky

spring field
#

MSE of the variance? like how far off the predicted variance is from the true one? isn't that just square of their difference? not like you can have multiple variances, can you?

past meteor
#

In your niche it's very true / important but I'm curious how many PhDs can answer this coherently. That doesn't make it right though

#

But you get my point

wooden sail
spring field
wooden sail
#

you can also compute its MSE

past meteor
#

If there's one thing I've learnt is that if you stare at any concept long enough you start to see it's more nuanced than you initially thought

wooden sail
pearl marten
#

1

spring field
spring field
wooden sail
#

you could directly compute the variance

past meteor
#

Do you have an actual statistics course?

#

If so, they'll teach this

wooden sail
#

you could compute it through the MSE by doing a bias-covariance decomposition though

past meteor
#

Notions of what an unbiased estimator is, is kind of important ye

spring field
wooden sail
#

"small number good"

past meteor
#

Here at least comp sci got less meaningful statistics than us

#

Econometrics was the best course I took in terms of statistical modelling

spring field
# past meteor Do you have an actual statistics course?

not yet, I kinda dropped out of the ME course I was doing, cuz... reasons
anyway, I'm planning on going to a different uni now for sth in cs, ds, stats, ml/ai, applied math, not entirely sure quite yet, so I'll probably have stats, yes

past meteor
#

The economics side of it is very boring but econometrics teaches you all the pitfalls of modelling in practice

#

Many of my comp sci peers didn't take it and were a lot worse at modelling

wooden sail
past meteor
#

No ML course teaches you about heteroscedasticity, predicted vs actual plots, analysing residuals, omitted variable bias, multicollinearity, ...

#

It's just about ML models and nothing more

#

Which is a mistake imho

spring field
past meteor
#

Maybe multicollinearity is one that is touched on

#

To motivate why you need regularisation

wooden sail
past meteor
wooden sail
#

you activated my trap card

past meteor
#

I'm honestly bad at giving precise definitions of things related to math

#

I can give you a hand wavy one

#

The only context we spoke about it in the context of ML was with ill-posed problems

wooden sail
#

most people would mention the explicit ones like multiplying or adding a term that makes the problem easier in some sense

past meteor
#

Like a matrix where one row is a linear combination of another

wooden sail
#

fishing out a particular solution, changing the type of convexity, etc

past meteor
#

Something something about the rank

wooden sail
#

roughly anything that makes a problem easier by restricting the solution space

past meteor
#

By adding an additional regularisation term you can solve it uniquely

#

I feel like "what is regularisation" is a question you can answer in 5 different ways

#

Especially if you're asking for practical applications, how to interpret it and so on

wooden sail
#

i also didn't like it right now, i was gonna add "or changing its geometry"

past meteor
#

I could talk about L2 regularisation being a Gaussian prior in the Bayes world and so on

wooden sail
#

now define "capacity"

past meteor
#

You can add noise to all your input, that's regularisation

past meteor
wooden sail
#

that would be a restriction of the parameter space

#

making it low dimensional

#

(it's not the only kind, but that works)

wooden sail
past meteor
#

Ah yes hearing "Tikhonov" sends me back to the university auditorium

wooden sail
#

L2 and L1 don't actually do that though, those aren't restrictions. they change the geometry

#

that was my beef with it

#

if anything L2 reg increases the dimension of the parameter space to make it smoother

#

as zestar says, increasing the rank of the matrix

#

more degrees of freedom

#

yes but that's not a restriction 😛 as you pointed out before

#

you can explicitly do this though, e.g. with how you complain about needing to use rectangular matrices instead of low rank square ones

#

THAT is a restriction

#

an additive term changes geometry and promotes a behavior, but does not restrict it

#

that's exactly it. and the intution is indeed correct

#

but a restriction eliminates the possibility of something happening

#

i bring it up because a restriction is something properly defined

#

i would be surprised if you could in the ML context

past meteor
#

Why is early stopping considered regularisation

wooden sail
#

cuz we don't have a whole lot of guarantees for ML

past meteor
#

Is there any basis for it?

wooden sail
#

i recommend looking at fisher information (this is what i study), which measures the information models carry about their parameter spaces. it's related to entropy, but they're not the same

wooden sail
past meteor
#

Then the definition is reduced to "thing that prevents over fitting"

wooden sail
#

i would interpret it as wanting a low dimensional solution

peak ridge
#

hmm

#

im building the next big thing
and a big thanks to this server

wooden sail
#

overfitting means your parameter space has a structure that contains even the noise, which is higher dimensional than the original parameter space (remember the previous noise discussion)

#

early stopping would keep your parameter space lean before you go on to fit the noise, so something like using a subspace of the full parameter space of your model

#

(not necessarily a vector space)

peak ridge
#

sure

past meteor
wooden sail
#

the whole discussion on "dimension" is pretty powerful

#

linalg too stronk

#

yeah smth like that

#

dimension accounts for it automatically 😌

past meteor
#

It's 2024 and I still haven't learnt how to use stan

wooden sail
#

treating it like a manifold with charts of dimension <= d might work

#

there's a reason topological methods are hot stuff

past meteor
wooden sail
#

stop when d starts to increase

#

i think incorporating time is a problem tbh

#

referring to deciding after how many iterations you have reached a solution, not to time

#

you can make the analogy, but you may inadvertently introduce misconceptions on how the optimization process works

#

if you compose a function with itself 10 times, where is the time axis?

#

gradient-based methods are like fixed-point iterations. you compose a function with itself N times and hope you ended up at a good solution

#

choosing to attach the number N to time is an arbitrary choice

#

which is an arbitrary choice 😛

past meteor
#

Is there an equivalence for gradient descent with early stopping when you have a closed form solution?

wooden sail
#

time need not be how you parametrize the curve the fixed point iteration traced

wooden sail
#

like L2 or L1

#

in that case you can formulate regularization as a bias

#

lead me astray daddy, but do it gently

#

idk why i said that but i think it captures the sentiment

past meteor
#

It's kind of nasty though, not like L2/L1

wooden sail
#

this is fine, but one can make the argument that we use math to model things outside our standard intuition too

past meteor
#

I use it because it's simpler to implement and reason about

wooden sail
#

that motivates the abstract, axiomatic approach of maths from the last 100 years

#

it's more powerful

#

(and less intuitive)

#

idk about clearly, that very much seems to be your preferred flavor only because you studied something related to physics

#

there's a distinction between what is actually happening and how you choose to interpret it. the way you interpret it may bring limitations and introduce misconceptions

#

certainly

#

a lot of momentum techniques, as the name implies in the first place, are studied exactly as you say

#

i think some of the popular interpretations of nesterov acceleration have something to do with damping of a spring or something like that

#

just as there are other studies that use none of that

#

yeah needs some statistical flavor on top

#

that's essentially it, yeah? you have a network that spits something out and then you evaluate it in a loss function that maps to R

#

so the composition of network and loss function is a functional acting on the inputs

#

but then you have a chicken and egg problem since calc of vars exists separately from physics 😛

#

it's also there if you include the data, since now you have to include expectation operations

#

every calculus student: wdym integrals are easier than sums

#

hilbert spaces are already directly connected to ML

#

R^n and C^n are hilbert spaces

#

the complex number part is usually kinda trivial, since you don't need the structure of C^n. cost functions f:C^n -> R are anyway not complex analytic

#

so you study them by isomorphism to R^2N (wirtinger calculus or splitting real and imag)

#

not really "larger" anyway though

#

since complex floats are internally represented as 2 float 64s

#

so internally what you have in the computer is also R^2N with special multiplication and addition structure

smoky robin
#

I have a question and for which I will have to post a picture is that okay

spring field
#

sure

latent girder
#

Hi, is this any good? Feel free to recommend, thanks. I just recently finished a basic python course and wanna jump into these stuffs. Im a complete beginner btw

image.png
spring field
#

might wanna practice Python for a bit first

latent girder
#

Alright thanks

lapis sequoia
smoky robin
#

My task is this

#

The only question I have is that by attributes it means column?

spring field
#

ngl, but features and channels mean roughly the same thing to me, if you could provide some additional context, like that last task mentioned or a sample of the dataset or sth

past meteor
#

If you have a multivariate time series you have n channels over t time points. Each channel has a measurement for each t ⋹ T. If you "flatten" it and give it to say a traditional ML model you have T * n features

spring field
#

so it's 6 params each embedded in 38 dimensions? over time

#

sth like
(batch, sequence, features, channels)
pithink

lapis sequoia
#

why is scipy.optimize never used? Are there just better methods? I know hyperparmeter tuning is better, but, yeah.

past meteor
#

I use it infrequently

lapis sequoia
#

optimization. I remember a couple of years ago, I used it to optimize some residual for lasso regression. I do not know, ML is optimization heavy.

past meteor
lapis sequoia
#

I had to use it years ago in grad school

#

he wanted us to do it that way for some reason

wooden sail
#

scipy optimize will either use heuristics for the gradients and hessians, or require you to provide them explicitly

past meteor
#

Nothing wrong with that. It just requires a bit more steps than just using sklearn. I use scipy directly when I have to

wooden sail
#

you use them when your optimization problem is nice and easy to formulate explicitly

lapis sequoia
#

Hessians man.

wooden sail
#

with deep learning this is not the case, so you use something different

lapis sequoia
#

That is when there is more than one variable to optimize, right? I do not remember

wooden sail
#

no

lapis sequoia
#

what is it

wooden sail
#

hessians? or which part?

past meteor
#

How do they fit lasso again? Coordinate descent?

lapis sequoia
#

yeah, second order partials: [f11,f12] [f21,f22]

wooden sail
lapis sequoia
#

youngs theorem

wooden sail
lapis sequoia
wooden sail
past meteor
#

The last time I used scipy was when I handrolled some time series methods because I don't like most implementations 😦

lapis sequoia
wooden sail
#

well, if there is only one variable, that's all you have

#

the hessian is the jacobian of the gradient. in the univariate case, that's just the second derivative of the one variable

#

in more dimensions you get also the cross derivatives

warm pebble
#

can somebody help me make an object detection ai

astral prism
#

the best

warm pebble
#

ok but i want to create my own

astral prism
warm pebble
#

ok i tried to download yolo but i dont know what file to choose on github

astral prism
#

u gonna use use yolo to train ur own ai model

flat sigil
#

finally having actual success 🙏

image.png
astral prism
#

instead on github

warm pebble
astral prism
#

u will understand

warm pebble
astral prism
lapis sequoia
wooden sail
#

i don't understand your question

lapis sequoia
#

the cross partials, like, f11 *f22 -( f12)**2 < 0; if f11 < 0; then it is at a maximum, if f22 < 0; maximum as well, in order for the cross-partials to hold, the signs need to be known. And that is only know by knowing if f12 = f21 or something

wooden sail
#

what one looks for is whether all of the eigenvalues of the hessian are positive or negative

lapis sequoia
#

f12 does not always equal f21

wooden sail
#

no, not always, but many functions you deal with in optimization do work this way

#

particularly under the condition i mentioned above

lapis sequoia
smoky robin
#

My model is giving me a recall score if 1

#

Yet other methods (accuracy, precision, f1 score) are showing accuracy of around 95

past meteor
#

That's very possible

#

Recall means that you classified all true positives as being of the positive class

smoky robin
#

Is that not over fitting of data?

past meteor
#

But you can have false positives, which is impacts your precision, accuracy and f1 score

#

Not necessarily

#

Imagine your data is 50/50 distributed between + and -

#

In your case you have 51/49 for example

smoky robin
#

It is 45/46

past meteor
#

All those that are + were classified as + but you misclassified a few extra as + that should've been -

#

then this shouldn't be too hard to figure out, look at the data 🙂

smoky robin
#

I get it a little bit

#

Thank you very much

arctic silo
#

datacamp it worth for learning data science and practising projects

serene scaffold
flat sigil
#

is it bad that the average minimum fitness of each generation is staying relatively constant?

image.png
lapis sequoia
#

Topics in ML that are the most advanced, what would you guys/girls, say?

lapis sequoia
flat sigil
flat sigil
#

Thx. After each training generation I just wrote the max mean and min fitness of that generation to a SQLite db and the used matplolib to plot it and scipy to make the smoothed curve

#

lol

#

Honestly I have never used dataframes before.

lapis sequoia
#

Any of you into Game Theory? I heard it was used in RL. Kind of obsessed with Game Theory and IO. Like, I have heard RL borrows from Game Theory, how?

hollow escarp
#

ANy ideas how can i optimize onnx model for arm64 architecture?

real phoenix
#

I want to webscrape IMDB top 1000 movie subtitles to classify them using NLP. Any suggestions as to how to proceed. There are permissions issues I am running into. I'd also be ok with using a dump of these files. Thank you.

hollow escarp
river cape
#

Hi guys does the word2vec need sentences or is it fine even if we feed only the words?

past meteor
# lapis sequoia Any of you into Game Theory? I heard it was used in RL. Kind of obsessed with Ga...

I did a bit of game theory in my bachelors. I think they're quite different because the classical RL setting just has a single agent whereas game theory is more of a multi-agent thing. When you have multiple agents there are game theoretic insights.

Another way you can look at it is from this perspective: in RL you have to learn the optimal policy and game theory actually gives you an answer to that question in the form of a nash-equilibrium (wherever present)

arctic silo
left tartan
left tartan
hollow escarp
left tartan
teal lance
#

It is possible to use lightweight charts and yfinance together ? 🤔

humble mesa
orchid forge
#

oh god, i just downloaded a dataset from kaggle, to make a project out of it. im not able to import a simple excel file

humble mesa
#

May you share the code?

orchid forge
#

sure hold on

humble mesa
#

dw take your time

#

I would use pandas if i were you

orchid forge
humble mesa
#

Is the file name correct? (Try removing the whitespace of your filename)

#

/ is the file in the current working directory

#

On my end works, considering the correct file path

image.png
orchid forge
orchid forge
humble mesa
#

Okay, is this file in the same directory as your notebook

orchid forge
humble mesa
#

don't forget the !pip

orchid forge
#

so everything is good with my file name?

humble mesa
#

And if you have a virtual environment, use python -m if you are inside your shell

orchid forge
#

there's no white space?

orchid forge
humble mesa
#

File name should be fine, just wasn't sure how pandas reacts to whitespaces xD

orchid forge
#

oh

humble mesa
#

This is bad, since some libraries might crash with different versions

#

so we use venvs (virtual environments).

Those are isolated environments where all libraries for your project are separated from the base python and other project environments

#

(At least this is how I understood it)

orchid forge
humble mesa
#

idk if that is the real technical answer but it has worked on my end since then 🙂

#

does it work now?

orchid forge
humble mesa
#

Hm ok.. wait let me check something really quick

orchid forge
#

k

humble mesa
#

You are on windows, right?

orchid forge
#

FileNotFoundError: [Errno 2] No such file or directory: 'salaries (2).csv.csv'

orchid forge
humble mesa
#

maybe it has to do with this (idk only an educated guess)

orchid forge
#

oh k lemme check

lapis sequoia
# past meteor I did a bit of game theory in my bachelors. I think they're quite different beca...

Game Theory has sequential move games(like chess) when you have to wait for the player 1 to move first and the second player acts based on the moves that the first player makes. Simultaneous move games are when both players act without knowing what you other player is doing. These games have a game table(most common game). Pure strategy Nash-Equillibrium is when both players cannot have a better pay off based on what the other player(s) has done. There does not have to be one Nash-equilibrium, there can be no Nash equilibrium and more there can be more than one pure-strategy Nash equilibrium. This is just simple game theory groundings. Game Theory becomes nonsense hard when it comes to Auctions. Auctions are extremely hard. I regress. Nash-Equillibrium in sequential move games is 'Sub-Game Perfect equillibrium'. There is first and second mover advantage. And, Sequential\simultaneous move games can be combined. There are also mixing games, mixed-strategy Nash equilibrium, Bayesian Nash Equilibrium, signaling games(hard), Infinitely Repeated prisoners dilemma(my favorite), Bertrand pricing games, Cournot games, Voting Games, Incentive Design(principal-agent), Collective Action Games. 'Agents' are players. I do not know. My friend is in RL and would ask me for Game Theory advice. Game Theory is incredibly underated and it is not a joke.

orchid forge
# humble mesa

do i have to add the file name right after i have pasted the whole path to that file?

humble mesa
#

So you mean something like this C:\path\to\ file (2).csv ?

orchid forge
#

ya

humble mesa
#

There is no space after the \ afaik

orchid forge
#

nope

humble mesa
#

damn

#

Ok let me google

orchid forge
#

im so sorry, i feel so embarassed

humble mesa
#

nah dw

#

We all have been there

  • I am neither an expert as well. Literally in my 2nd semester xD
orchid forge
#

omg i did it

humble mesa
#

how??

orchid forge
#

i got it done

humble mesa
#

Awesome

orchid forge
#

thanks

#

im just a stupid beginner im the only person with with dumb questions

humble mesa
#

Nono you are all good 🙂

I initially learned Java before my uni and now I am in my 2nd semester of python so I am new as well

#

What was the fix btw?

orchid forge
#

i made a stupid problem, i didnt add 'r'

humble mesa
#

Happens xD

I literally got roasted by Java sometimes for not adding a ; in line 42

#

see it positively: you will never forget the r 😄

orchid forge
#

ya

#

lol

misty shuttle
#

how do i proceed with ML if i've learnt numpy, pandas, matplotlib and scikit learn?

humble mesa
#

What helped me was doing a data visualisation project

#

I analysed the top 1000 streamers world wide and ran some statistics on a public dataset.

#

So my go-to answer would be:

1. get some dataset
2. Try to visualise the data 
3. Upload to git
#

btw i like matplotlib as well, however plotly express is perfect for interactive elements

misty shuttle
hollow escarp
serene scaffold
hollow escarp
#

!pastebin

arctic wedgeBOT
#
Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

hollow escarp
# 
      if not frame_queue.empty():
        frame = frame_queue.get()
        # cv2.imshow("Frame", frame)
        height, width = frame.shape[:2]
        processed_image = prepare_image_input(frame)
        plate = detect_closest_license_plate(
          session, processed_image, height, width, app_logger
        )``` Here is how i trigger that function for camera stream
#

How can I send you my model in onnx format to not get message from @arctic wedge ?

#

@serene scaffold what else do you need?

lapis sequoia
#

does anyone have a really good guide to NLPs? I have been working on some, but I just want a good guide

#

one that is not outdated

serene scaffold
lapis sequoia
#

ok, so like, countvectorize,tfidvectorize, when one starts the train/test split, I do not know I get this one error all of the time. I do not know if it is because I used tfid instead of countvectorize or vice versa

serene scaffold
lapis sequoia
serene scaffold
lapis sequoia
#

I will figure it out. It always heppens during one step. I do not know it is confusing me

serene scaffold
lapis sequoia
lapis sequoia
# serene scaffold I'm on mobile, so I can't dig in more at the moment. You might make a reproducib...

it is this. Everytime, Sorry, not trying to bother you, this is the error,and this is the snippet of code I am talking about: raise ValueError(Errors.E1041.format(type=type(doc_like)))

ValueError: [E1041] Expected a string, Doc, or bytes as input, but got: <class 'list'> def lemmatize_reviews(df):
df['Quote'] = df['Quote'].apply(lemmatize_text)
return df

from nltk.tokenize import word_tokenize

def lemmatize_text(text):
doc = nlp(text)
lemmatized_text = ' '.join([token.lemma_ for token in doc])
return lemmatized_text

def do_tokenization(text):

token_words = word_tokenize(text)
return token_words

df['Quote'] = df['Quote'].apply(do_tokenization)

from sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer

tfid = TfidfVectorizer(preprocessor=do_tokenization)

df['Quote'] = df['Quote'].apply(lemmatize_text)
df['Anime'] = df['Anime'].apply(lemmatize_text)

cv = CountVectorizer()

spring field
arctic wedgeBOT
#
Formatting code on Discord

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

For long code samples, you can use our pastebin.

frosty spear
# 
print("Code")
sand bloom
#

Hii, can i ask for simple help on a Collaborative filtering question?

serene scaffold
small ore
#

" implemented sklearn library for cosine" ???? It sounds like you implemented that class in sklearn

#

As in the source code

sand bloom
#

yes sorry

sand bloom
#

Hi no worries sorted my issue my question was just dumbly asked

spring field
#

PER in RL sort of seems similar to DML and triplet loss where you try to get the furthest same class and closest other class to your current thing you're checking to sort of push away the other class and bring the same class closer
similarly PER should enable both retrieving low score actions in states thus making it less likely to make such actions in the future (pushing it away) and bring actions that had greater rewards closer (if making an analogy for triplet loss)

stray tide
#

Hey everyone! 😊
I'm diving back into programming after a bit of a break and planning to create a stock market prediction ML model. I've got some basic ML knowledge from Kaggle, but I'm looking for some buddies to learn and build this together. Programming alone can get pretty dull, and having friends makes it so much more fun! If you're interested, hit me up! Let's make this project awesome together!

stiff urchin
#

what concepts do i need to learn in calculus for machine learning? Already learnt integration and differentiation, what else?

stiff urchin
#

do i need to learn each and every topic for ml?

#

not backend engineering, ig i would need to learn different different languages for that.. currently im planning for research

wooden sail
#

research of what kind

#

the applied approach requires less maths, but stuff like designing new architectures and studying their properties requires a fair amaount of math

#

all of those are made up terms trying to keep up with the tasks people end up doing at work, idk how meaningful it is to distinguish them strictly when companies don't

#

i don't think the experience will be much different in most places

toxic mortar
#

do you have phd? @final kiln

#

Have you looked what europe-based ml programs for msc/phd can offer?

stiff urchin
#

actually, im still in high school 😅 currently want to learn ml as a hobby so in future i can become data scientist. and im even preparing for one of the toughest exams in the world (jee; you might know about it), i believe it consists of some of the concepts of calculus, it'll help building my foundation and then I'll start with advanced topics

toxic mortar
#

Does it makes sense to do phd in ml if u are not aiming for r&d roles?

#

I will be soon getting into it and I'll share what I hear/find w u

past meteor
#

The term data scientist is so dead

#

ML engineer will perish soon as well

#

Nobody, it's going to be a game of revolving chairs

#

Data scientist nowadays means "does anything with data"

#

ML engineer will soon mean "does anything with ML", including calling openai's APIs

#

Just wait, I saw the same thing gradually happen to data science

#

Well, nowadays it just means "makes dashboards in powerbi" anyway 😭

humble mesa
past meteor
#

Yeah, I'm working exclusively on deep learning right now but I'm not sure if there will be any other project like mine after it's done

dusty talon
#

hey uh im new to ai i tried to make a classifier ai im not sure why its saying ball is a verb though any idea? i have a feeling its lack of data because im using not a lot of data but ??
https://paste.pythondiscord.com/YGAQ

humble mesa
#

JEEEEEEZ try a bow approach

#

I mean I am in my 2nd semester. I honestly haven't worked so much with that but what I would do is take random text

build bigrams

dusty talon
#

alright let me try tysm

spring field
#

pop!

edgy elm
#

Guys i got a question about some confussion matrix why my image doesnt show the number? while i print it have the number shown

#

like this

image.png
spring field
#

hard to tell without the code

#

yeah, I was thinking about its applications in RL as well

misty shuttle
edgy elm
orchid forge
#

hola

#

im making a project on this type of data

data1.png data2.png
#

can some one help me with what type of questions i should come up with to make a project about it?

uncut plaza
orchid forge
uncut plaza
#

is this a personal project or related to school?

orchid forge
#

personal project

uncut plaza
#

okay that's awesome, keep working on it

orchid forge
#

thanks man. on to it haha

analog bolt
#

So making an AI that does one thing, e.g, guessing a country based on its shape, is relatively simple, but how much harder is making an AI that is able to do 2 things, e.g, can recognise countries based on shape, but can also recognise countries based on their flags?

torpid dune
#

hi everyone! i was looking for an all in one ml ecosystem, as lightning studio AI, does someone use it? How about speed? do you have any better recommendation for realtime inference?

uncut plaza
analog bolt
#

How much harder would it be to create a model that can do both?

uncut plaza
#

depends on what your definition of hard is

#

like i said it should be the same

spare briar
torpid dune
#

context: im building a multi agent with langgraph.
But this multiagent contains multiples models. One of that, is a Vision model.
The other ones are simples llm inference endpoints.

The vision model, currently is an NVIDIA endpoint, but it takes too much time like 2-3 seconds. Im looking for deploy a self-hosted vision model to try to reduce the inference time.

spare briar
#

i would recommend tensorrt + triton inference server

#

prod ready, fast, maintained well

torpid dune
spare briar
#

yes we use grpc and have had good experience

earnest finch
#

hello everyone, i wanted to learn reinforcement learning so came here to ask suggestions for good resources available online
i have completed my college math and have a decent understanding of statistics, lin alg and calculus.
i have worked with simple ANNs like making it from scratch (shoutout to andrej karpathy for it). unfortunately i couldnt find any videos from him which teach RL so wanted your opinions on where else/who else can i watch for learning how RL's work from scratch. im fine with papers as well (as long as they dont include any complicated math notations/vocabulary 😅 as i still consider myself to be a beginner)

spring field
past meteor
earnest finch
#

ahhh will check them out, thank you

past meteor
#

The devil is in the details and just reading the text was not enough for me to understand the nuance

#

Especially when you start comparing on-policy vs off-policy methods etc

earnest finch
#

thank you so much

past meteor
#

It was pretty fun. After writing this stuff from scratch (just numpy + matplotlib) everything became clear 😄

image.png
spring field
#

I'm doing policy stuff now... it's a bit of a confusing topic to say the least and I don't understand why it's not converging waaaaaaaaaahhhhhh I mean it is, so maybe the issue is more with policy gradient not being efficient enough on its own shruganimated

past meteor
earnest finch
#

maybe i will understand the magical words used once i read the books 😄

past meteor
#

there's a very logical trajectory to all of this

earnest finch
#

im assuming i start with full pdf link?

past meteor
#

beginning with policy gradient methods would've confused the heck out of me

#

Also, many of the algorithms are proven to converge under specific conditions

spring field
#

I mean, I didn't begin with policy gradients, I'm there now

past meteor
#

like, on-policy + tabular => converges

#

off-policy + function approximation => can possibly not converge etc

spring field
#

nvm, I'm gonna just go and read that book, lmao

past meteor
#

policy gradient should be on-policy though iirc

spring field
#

I'm just currently going over all those methods really, I think aim is to get familiar with a bunch of network types and then specialize later? the course I mentioned a couple days ago

river cape
#

How do you handle imbalance data?

spring field
#

tbf, from a learning perspective, making multiple passes over learning material, going deeper and deeper on each iteration is a better method than just going one by one and trying to have a deep understanding immediately

spring field
past meteor
#

if that's your distribution, that's your distribution

spring field
#

squint hmmm

river cape
past meteor
river cape
#

Should I take 20% yes cases only?

past meteor
#

It has a logical structure that makes RL seem coherent, if you go method by method (as is actually not a bad idea for traditional ML) you're kind of losing out on that

spring field
past meteor
#

oooooh

#

yeah ok that makes sense

#

then I agree

spring field
#

but if I'm gonna be doing some more RL afterwards, I'll definitely check out that book, bookmarked it

past meteor
#

If your course touched on RL it's probably a really good one

#

Maybe we should pin it?

#

As a nice "overview" type of course

spring field
#

it's, uhh, a private one so to speak, but it is good, yes 😁

past meteor
#

fair enough! 🙂

lusty relic
#

Is levenshtein good option for spell checking if i have sequences of words in database for search recommendations?

past meteor
#

this is true but typically not the case, at least in applications where people talk about unbalanced data

spring field
#

alright, suppose it's a classification task, you'd want to handle class unbalance there for sure, right?

past meteor
#

There's also the misconception that the method will just ignore the minority class in favour of the majority

#

That's not necessarily true

past meteor
past meteor
spring field
#

I can understand that specific example

past meteor
#

yeah I agree on it too but the thing is

#

I wouldn't want to mention it because I don't like talking about downsampling/upsampling because that's what people like doing which is just the wrong thing to do in many cases

#

For instance, I did a course on biometrics (making models for fingerprint, retina, face, ... detection) and we didn't downsample or anything funky a single time

#

You can imagine how unbalanced these datasets are.

#

We just used common sense, less sexy ways to deal with imbalance

spring field
past meteor
#

DML?

spring field
#

Deep Metric Learning

#

how face detection works and stuff

past meteor
#

yeah, that's one of the things we did

#

but there's many other ways

spring field
#

AVE?

#

ah, these, we did cover them too at some point

past meteor
#

Looking back at it, we did eigenfaces, fisher faces, local binary patterns and deep metric learning

#

I also did deep metric learning for retina detection

#

well, classification/retrieval

#

damn, sift features are literally stone age tech now

image.png
past meteor
#

It's your eye

#

they're unique identifiers, even between twins

#

ah shit

#

true

#

like this stuff

#

was a cool course in hindsight, kind of like applied computer vision

#

the one I sent?

#

it's legit ye but these were the originals, before preprocessing

image.png
spring field
past meteor
spring field
#

oh, I just don't see the correlation, lol

past meteor
#

hmm?

spring field
#

is it like, unrolled?

past meteor
#

yes

spring field
#

idk, looks a bit more than just split in half

past meteor
#

the code I copy pasted is nasty

#

but it works 🤷

#

Whenever people at work talk about "neural nets are too heavyweight" I show them stuff like this to show how finnicky pre NN comp vision was

image.png image.png
#

Way more effort for worse results 😉 😄

spring field
#

mmm, cuz I'm thinking this

image.png
past meteor
spring field
#

idk, the pupil is weird anyway

past meteor
#

fsr I didn't ask as many questions about this as you guys are, but that's a good thing on your part

#

I just accepted the preprocessing had processed

#

I actually haven't used vision transformers

#

CNNs only 👴

spring field
#

I did not need to know that...

#

I saw some performance graphs and that did seem to be the case
though seeing where they pay their attention is quite fun in ViTs

#

and I assume ViViT probably does beat a CNN, cuz a CNN can't really do frame processing, I mean, ig you could try do an RNN + CNN pithink

past meteor
#

This was a lot of fun. I made a novel method that can add image specific noise (output of a NN) to turn any image into an image neural net 2 to trick it into believing it's an airplane.

image.png image.png
#

The noise is very visible but that's because I didn't tune my loss function, I used a basic heuristic

spring field
#

ah, the likely future of AI cybersecurity pg_rofl

past meteor
#

I wonder if something similar can be done to open source transformers

#

You need whitebox access to the gradients, which you absolutely do for llama etc.

lapis sequoia
past meteor
#

I should do more ML projects. I never do (aside my actual job)

spring field
#

same here (except I don't have a job yet...)

past meteor
#

Yeah, I think Matiss is talking about video

#

RNN+CNN is truly a thing for those applications yeah

#

Type of role where I suspect you'll be doing dashboards 24/7

#

I applied, nothing to lose

#

Want to know the secret of the high callback ratio? (I suspect I'll get a callback for this one very soon)

#

Here most ML people can't really really code well to the extent they can put things into prod

#

yes

#

If you can, and have demonstrable experience doing so you're ahead

#

Notice how using version control is a nice to have 😭

image.png
#

I see it with some of my colleagues. One does all his "development" inside a browser IDE in our SaaS. Doesn't version control anything "because it is saved in the browser"

#

Brilliant guy, but that's reality

#

I detest browser IDEs

#

Made me really dislike databricks

#

I need my vs code with my very specific colour template, keybindings, ...

#

I sync it between work and private

#

I just want them to give me a way to SSH into their environment

#

and code with vscode

#

ah, then it's np

#

I wouldn't do Gen AI focused roles ngl

#

The risk is losing touch with actually training models and the difficulties with deploying them

#

If you're just doing it with existing APIs you're effectively a backend SWE

#

Just my 2 cents

#

yeah, this stuff is actually engaging

#

because people (incl. myself) don't really understand it

#

It was the same issue with my current job. All our projects were too abstract for regular people with no tangible use case

#

"Spinal deformity detection using wavelet features", "glucose prediction models for people with type 1 diabetes", ...

The only thing we made that people could understand was some IoT computer vision thing

#

It had a UI, clear output etc

#

That's a factor as well

#

I'm gonna let the NLP train pass me I think

#

Haven't done enough in it and for some reason I can't be bothered to either

#

Just did the info retrieval thing in uni (which is the foundation for RAGs etc) that didn't do the actual NLP course

#

The cost of training, deploying, rlhf,, ...

hollow escarp
#

Here is my model converted to onnx format

craggy patio
#

!paste

arctic wedgeBOT
#
Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

hollow escarp
hollow escarp
#

I dont think that pasting model to paste bin makes sens

#

Also im really glad for any ideas how can i speed up my model

craggy patio
#

oh that's actually really funny mb

#

I did that for myself

#

I was pasting code in the rust discord server but it was too big

#

I needed the link for pastebin but I didn't know it off the top my head so I went to a random channel and did ! paste

#

sorry for the confusion

left tartan
spring field
#

ViViT uses transformers, lol

torpid dune
spring field
#

I was just wondering if CNN + RNN could do sth with videos

trail whale
#

Ok I have a long and convoluted question for someone with a bigger brain than me:
So I know there are tons of GPT/LM services online that are free where I can send a prompt and the thing returns a response. And I know the way this works is it tokenizes my input, vectorizes those tokens, sends data into a neural network, and it outputs all the possible tokens' confidence values, or how sure it is that every token proceeds the last. My question is, I'm wondering if there's an API or a cloud-based thing where I can send a string as input, and the output is the confidence values for the next token in the string. For example, if the input is "The sun is", the output could be [["hot", 98], ["big", 95], ...]
TLDR: Is there an API where I can get the list of the raw output nodes of a GPT?

agile cobalt
#

you may as well just run Llama or Phi locally instead

much of the time the tokens are not entire words, but rather fragments that may make no sense on their own like "un"

spare briar
#

openai completions api exposes logprobs but at most top 5

#

wonder if this is related to message size? or to make teacher-student distillation more difficult?

spring field
#

alright, ig a TokenLearner specifically probably cannot, but the general idea at least

#

network learns how to split words/sentences itself

agile cobalt
#

TokenLearner itself appears to be specifically made for working with multi dimension data, so yeah probably not a good idea to use it directly on text

spare briar
#

tokenizers are typically trained separately (most commonly by byte-pair encoding or unigram model https://github.com/google/sentencepiece)

there is some work like this https://arxiv.org/abs/2106.12672 on tokenizers learned e2e with pretraining but it isn't used in sota LLMs

arXiv.org
Charformer: Fast Character Transformers via Gradient-based Subword ...

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-...

agile cobalt
spring field
spring field
agile cobalt
#

I don't know in enough detail to explain it

spring field
#

fair enough

orchid forge
#

can someone give me some EDA tutorial ?

#

good one

#

course i mean

past meteor
bitter ibex
#

Hi I am working on a project using sklearn and i am facing an error in my final project.
If anybody can help me solve my problem , that would be greatfull. Just ping me in the dm.
Thanks

odd meteor
orchid forge
past meteor
#

yes, make an account and check it out

left tartan
pliant heron
# 
# Screen time analysis is the task of analyzing and creating a report on which applications and websites are used by the user for how much time. Apple devices hve one of the best ways of creating a screen time report.
import pandas as pd
import numpy as np
import plotly.express as px
import plotly.graph_objects as go

data = pd.read_csv("Screentime_App_Details.csv")
# print(data.head())
# print(data.isnull().sum())
# print(data.describe())

figure = px.bar(data_frame=data, x="Date", y="Usage", color="App", title="Usage Graph")
figure.show()

PLease someone help me in fixing this error
i have tried deactivating my anti-virus but still the issue persists

image.png
summer anvil
pliant heron
#

thankyou

summer anvil
left tartan
pliant heron
left tartan
#

In a .py or .ipynb?

pliant heron
worn adder
#

i wanna get into this sector, ik basic python , can anyone tell like what exatly it is and the roadmap? thank you

pliant heron
#

i m looking for this output

image.png
left tartan
#

Yah, so you probably want to just write the image or figure.to html

summer anvil
#

then use simplehttpserver or smt idk

pliant heron
#

how to do it

left tartan
summer anvil
pliant heron
#

thanks billy and hoang

left tartan
#

The explore chapter, if that's what you mean, yes

#

Its a very thorough document

#

Yup, exactly. And practitioners get frustrated with the recent trend to throw ML at data and forego EDA

past meteor
#

Too many people never look at the data

#

When I say look I just really mean, sample some of it images, records, documents, whatever it may be and read it

#

no graphs just looking

#

From the questions that come here that's often implicit (not looking)

#

As for EDA

#

I've said it many times, I think you should avoid doing ML and your first line of defence is a good EDA such that you could find heuristics that solve the problem sufficiently

#

This is usually a thing with tabular data only imho

#

Agreed but when it's about EDA I guess the assumption is that it's tabular data

#

Or at most time series

#

There's degrees to it though. One of my colleagues is a bio-engineer more analytics focused (+ has more domain knowledge in our project)

#

The EDA's he does are things I'd never dream of

#

They have tons of depth

#

Sometimes they're complex in a bad way but still, loads of depth

night kettle
#

Guys , please if someone can help me plot my data just like in the graoh below

IMG-20240528-WA0007.jpg
stray tide
#

I am making a stock market price predictions how to give the time stamp feature to the model

#

i have given there not sure if it is of any use

#add day of  features
df['day_week']=df['Datetime'].dt.dayofweek
df['day_month']=df['Datetime'].dt.day
df['month']=df['Datetime'].dt.month

df['time']=df['Datetime'].map(lambda a:(pd.Timedelta(a-pd.to_datetime('2023-1-14 9:15:00+05:30'))/pd.Timedelta('5m'))%(24*12))
#

!code

spring field
#

I don't think Lisan here is gonna like my suggestion (pg_rofl), but... RNNs are usually what you'd use for time series data such as this and you'd simply provide the data in a sequence
probably could use transformers for it as well since they have positional encoding, but idk
other than RNNs, there are a bunch of other time series thingies in ML like ARMA, ARIMA, SARIMA, SARIMAX and whatnot

dusty talon
#

Is it better to make my own ai from scratch? or use pre built libraries already im still new to ai, and im not sure whether i must go down to learn, how LLM, RNN.. etc work, and then program my own from scratch or should i use a library like, tensorflow , torch or Hugging face???
Everything is still a bit confusing for me, i do understand python so i guess that would be my starting point i do suck at maths though.

Im kinda looking for a starting point, and where to go from their.?

stray tide
#

Once you get a hang of it you might try coding it on your own but not sure if it will be ever needed in a practical scenario I might be wrong that what I know

spring field
#

but of course, it's wishful thinking at best, but if you believe there are patterns to find, might as well I guess, besides it's not a terrible learning experience anyway

spring field
#

reminds me of that time monkeys throwing darts beat the market... (I just found an article on Forbes and it's actually a bit more than that, the explanations and stuff, it's quite interesting)

#

one can hope waaaaaaaaaahhhhhh

frank dagger
#

anyone familiar with assigning weights to datapoinst in a dataset and then training these weight in parallel to the original dataset, in order to determine which datapoints that are more reliable when sampling together a prediction?

hollow escarp
#

Or recognition process with out Using GPU

narrow tiger
#

any generative AI videos u guys recommend to add custom knowledge to models/ create agents get used to basics thanks

past meteor
#

There's been a lot of research on this

river cape
#

Hey guys I have just finished these concepts in NLP:- Text preprocessing , One hot Encoding , Bag Of Words, N -grams , TF-IDF , Word2Vec and POS tagging . What else should learn to improve in NLP?

past meteor
#

It is

#

Oh, I was talking about time series in general

#

Not stocks

hollow escarp
#

Any ideas what additional img transformations should i make to make this img easier to read by OCR?

image.png
#

Thats img before any preprocessing operations

image.png
#

Im using easyocr

#

For OCR actions

hollow escarp
#

How to do that?

#

Thx, let me try it

narrow tiger
#

can irun mistral on 1660ti?

hollow escarp
#

I think thats not that easy task, im making OCR for licesne plates so i would need a lot of time and resources for creating good model

narrow tiger
#

have you tried creating this?

#

it maybe if you can make it see where liq is

#

and price is fractal

#

it might work to some extent

#

using smart money concepts

hollow escarp
#

I will try it, im not really good in training data sets most of the time i used already created by someone

narrow tiger
#

cool can u suggest any good read on 'fractal time series'

hollow escarp
#

But for sure i will try it right after connectedCompontents function

narrow tiger
#

i never said u should use numbers for this

#

price is fractal
by this i mean price action repeats on different time frames
so if u can make a model that can learn different patterns on different time frames and can alsoo see where most of the liq is

#

it might be possible to predict direction to some extent from higher to lower time frames

#

ngl i have no idea what that means
but as a trader when we say price is fractal it means
it behaves same way on different time frames

#

liq runs same happens on all time frames

#

tag buy side and then to sell side imediatly

#

buy side liquidity == people who's buy orders got filled from being short(liquidated)

#

what do u mean by "it is not present in the numbers "

#

bro u don't have to beleive me
but u should look up 'smart money concepts'
that's how people understand what market is about to do

#

and i been trading for 3 years now

#

haha people do say just by looking at raw price action you can predict if something at scale is gonna happen

#

going by that defination you can't really predict the market ever no matter what you do

#

but u can always use probability and risk management to make educated guess

#

nope u can't
the only way to 100% accurately predict is by knowing what everyone involved is gonna do with certiniity

#

but try delta neutral stratgies you are good with math u can come up with somethiing like that i am sure

#

there is theoratical part and there is what actually happens

#

news has little effect on markets

#

people who are sources of the news usually have already taken their sides of the bet so news is almost always priced in

#

well let me know what you some up with

#

try paper trading
i have built alot of trading bots most of them only make money under some specific conditions or just loose
thanks for the resources hopefully i'll learn something new

hollow escarp
#

Okay i think trying to get that high resoults with OCR doesnt make sens, @final kiln could you share some more details about training UNET model?

#

What dataset did you use

#

There is no such repo

#

Label you mean the location of license plate or actual text?

image.png
#

Nope

#

What do you mean by that?

#

Okay i could try to generate such data

#

Okay assuming i have a set which includes every letter in 40 random license plates from the set above how should i train that data? As i told you im really bad in traning stuff

#

Oh okay

#

Okay im glad for your advices for sure i will try to train such model

#

Okay thxx

fallow coyote
#

Im currently doing a guided project (stock price predictor). I'm continually commenting throughout my code to use as a guide when I go onto make a solo ML project. Is what I'm doing useful trying to improve my ML programming skills? What can I do to improve them further?

dusty talon
hollow escarp
#

Im wondering about one think, because imgs which are provided in the data set are "to good" what i mean by that: I mean that my camera doesnt record in that high quality the imgs in the data set

#

Also i know that my camera setup could be better so im thinking also about first trying with different angle

river cape
#

Hi guys is advanced NLP known as Gen AI?

hollow escarp
#

Okay

river cape
#

No idea I just finished ext preprocessing , One hot Encoding , Bag Of Words, N -grams , TF-IDF , Word2Vec and POS tagging. So i was wondering what to do next

#

I am sorry but I dont have any idea about what should I learn next

serene scaffold
#

Technologies like ChatGPT are interactive, generative language models. And interactive generative language models are in vogue at the moment. But they aren't the only application of language models. Or the only AI technologies that are both interactive and generative.

river cape
#

I am sorryyy🥲

river cape
serene scaffold
river cape
serene scaffold
river cape
serene scaffold
#

this isn't a serious suggestion.

serene scaffold
river cape
serene scaffold
river cape
serene scaffold
river cape
serene scaffold
river cape
narrow tiger
#

how does a local running ollama model hold soo much info?

#

it is only like 4gb

#

by having info i mean
it is answering all the questions?
my understanding is LLMs just copy paste knowledge that they have seen before (they aren't able to think)
so how can a 4gb model answer all those questions

wooden sail
#

it's better to think of it as building a function that assigns tokens a probability of occuring together

#

no copy pasting is going on

narrow tiger
#

mb didn't mean literally copypasting

wooden sail
#

it's more like building a function that you feed some text into, and it tells you what it thinks comes next by assigning a probability to all the tokens it knows

#

and as for how much predictive power, let's do a quick calculation

narrow tiger
spring field
#

yep

wooden sail
#

with the distinction that the prediction is not deterministic

narrow tiger
#

ohh ok ok now i see how hullicination might occur

wooden sail
#

not all models have a random sampling effect

narrow tiger
#

this is magic 😂

spring field
wooden sail
#

at any rate: 4GB. these models often use 32 bit floats. we can fit 4*10^9/32 floats in that much memory

wooden sail
#

.wa s 4*10^9/32

strange elbowBOT
wooden sail
#

that's the number of parameters in the model

spring field
#

mmm, ig that makes sense, otherwise it would indeed be rather boring if it were deterministic

wooden sail
#

asking exactly the same question on different sessions should net you slightly different responses

narrow tiger
#

it kinda makes sense
i have only done very basic neural networks and the statistics did work
but to think that i can work for something as natural language to "guess" what comes next and still able to make a sentence which makes sense is unbelievable

wooden sail
#

whether it learns how to assign probabilities correctly depends entirely on the training data though

#

there are lots of cool experiments tricking language models into spitting out tokens in ways that make no sense

#

you can get stuff that doesn't make up words at all

narrow tiger
#

is there a basic grammar defined like it has to map token with real words

wooden sail
#

no

#

that's all learned

narrow tiger
#

so how does iit know what "Apoptosis" is

wooden sail
#

it doesn't

#

but in the training data, the tokens making up that word only appear in special combinations, so seeing that in a sentence in the correct context (with other tokens in the correct order) tells it to assign the upcoming tokens a high probability

narrow tiger
#

is there like a very basic LLM you can train yourself

#

to better understand the process

wooden sail
#

there should be, try scrolling up. i recall someone training a model of their own in the past few days

#

maybe matiiss, actually

#

i also remember that person getting random tokens sometimes

spring field
#

I mean, it makes sense

narrow tiger
#

like for ml i did an example for
[1,0,0,1] so had to fit the line by first 3 elements and train to predict the 4th

spring field
narrow tiger
wooden sail
#

i think i was thinking of this

#

maybe not, idk

spring field
#

that was an RNN

wooden sail
#

ah ic, i wasn't paying attention

spring field
#

I mean, I did attempt next token with GPTs as well, I just forgot to actually do the rollouts 😁

#

and also I implemented it like a translator...

wooden sail
#

that does cut the parameters from N^2 to 2N - 1

#

N if you use a symmetric kernel

#

we used a similar trick in a recent paper cuz a model required a few petabytes of memory otherwise

#

you give something up though. CNNs enforce shift invariance, which is something not all sentences have

#

could be interesting. there will definitely be a tradeoff between memory gains and accuracy as you play with the window size

hollow escarp
#

in the message pined there are some info about model

hollow escarp
#

Im using onnxruntime

#

They told me here that using onnxruntime is really good solution for deploying object recognition in dockers

#

i've had a problem with puting my yolo object detection to docker

#

and here told me to use onnxruntime for it

#

Oh i remeber that was to problem

#

The size

#

@final kiln

#

I need to deploy it to field devices, so it cant be that big

#

Thats the size of my docker now

image.png
#

And i know i can cut like 100MB more from it's compressed size

#

Nope, wait i will send you my docker

#

There is a bit mess here becasuse i was trying some different stuff with mutli platform imgs, but here you go:

FROM --platform=$TARGETPLATFORM python:3.11.9-slim-bullseye

ARG TARGETPLATFORM

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        libgl1 \
        libglib2.0-0 \
        gcc \
        libhdf5-dev \
        cmake \
        g++ \
        libffi-dev \
        pkg-config && \
        if [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then \
            apt-get install -y --no-install-recommends ninja-build; \
        fi

RUN apt-get install -y --no-install-recommends libssl-dev 

WORKDIR /app

RUN pip install --upgrade pip setuptools wheel && \
    pip install --no-binary=h5py h5py --no-cache-dir

COPY requirements.txt .
# COPY /wheels .

# RUN if [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then \
#     pip install onnxruntime-1.16.0-cp311-cp311-linux_armv7l.whl; \
#     apt-get install python3-scipy -y --no-install-recommends; \
# fi

RUN pip install --no-cache-dir -r requirements.txt --find-links https://download.pytorch.org/whl/cpu 

COPY . .

RUN echo "Building for $TARGETPLATFORM"

CMD ["python", "main.py"]```
#

What do you mean "no you did"?

#

You can cut a bit by using mutli stage builds

#

But it wont be much

#

No i followed the instaltion for CPU only

#

Oh you mean in old img

hollow escarp
#

Ye so i installed it with CUDA there

#

But even with out CUDA i had like 5GB or like 4.7GB of img

#

I heard about some model quantization but never investigated in subject

#

I tried bunch of ways but non of them gave me any improvements

#

Okay, so i should grab some controllers with GPU or start being AWS slave XD

#

Ye i know, im just joking

#

I will read that options really carefully, im really glad for all you support

#

Ye i know that

fallow coyote
#

I swear it always feels like im going back to square one with my ML journey. i see you all with somewhat clear aims to be in ML/AI/DS judging by the level of knowledge you all have, whereas I'm just a uni dropout doing this shit because it seems interesting, not knowing how much fucking work it requires with no aim in what to use it in

fallow coyote
#

I dropped out of 1st year. Got sick of the studying. Applying to apprenticeships (engineering and it). Idk wtf i want to do

#

Still waiting for confirmation if im successful or not. It just feels shit. When i have no aim or guide, i cant focus in doing something. Like with the ML/AI. I cant put the hours needed to be good at it if i dont know what im going to do in the future with it

#

Well its what i plan to do. Do an apprenticeship and then use that to get into a degree apprenticeship scheme. I just want to do something to pass the time and have it be beneficial for me in the future

#

I swear bro its like in constantly going back square one every few days

fallow coyote
#

It used to be my dream but when you spend so many years studying and then for it to not go anywhere and seeing your dreams collapse in front of you with barely anyone helping you out, it destroys and changes how you think about things

serene scaffold
fallow coyote
#

I applied to an IT apprenticeship with BT and an engineering apprenticeship with another company. If i do well, ill do a degree apprenticeship where i can get a degree through my apprenticeship (no need to pay for anything)

latent girder
#

is there any good interactive free data science tutorial/course? Most yt's are boring. A full of non-stop talking

#

oh nvm I mean just python, not the whole data science stuffs.

#

sry about that

fallow coyote
#

Ill see how it goes. Hopefully i get something back from them

peak ridge
#

@final kiln

#

how was the job interview?

#

on paper im millionaire

#

on paper valuation , my share price

#

you'll get the job

#

nice.

#

do u have LangChain knowledge tho?

#

as someone who hires,
I would say it's not about just interview
Some ppl interview to hire "actually" and some to reject, it's just matter of requirement and urgency, scarcity

#

for backup

#

they have options, but to be on safer side they i'll get back to u
after even sometimes 6 months there another hire when fucks up they'll get to u

#

that's sad.bad but reality.

#

oh, what was different?

Maybe i will try to make our company like this

#

what was so different in this Interview Vs Others @final kiln

#

can u give me a bit detailed insight, maybe i could apply it too

#

notice everything and please do lemme know with insights

#

so she took personalized interest in ur work and gave u more ideas and stuff

#

very very good, i do this but i'll take this more into consideration

#

crazy

#

where are u from ?

#

which country

#

oh

#

why?

#

why didnt u like the HR

#

(im trying to scrape everything that a developer like u thinks of a interviewer/a company, so i can maybe identify something wrong or apply something seeming to be useful)

#

ok, what kind of personality ppl
"developers" like you wanna work with

azure igloo
#

hi! sorry for interrupting, quick question. I was wondering how I could figure out how much memory capacity/processing power would be ideal for my AI model? i'm using a python wrapper called faster whisper to live transcribe voice into text and running it on my normal CPU isn't cutting it at all

peak ridge
#

okay, i get it.

azure igloo
#

okay

#

i'm eventually planning to move it to a raspberry pi

#

how would that work?

#

oh hm

#

me and my friends were planning to make a robotic dog with speech recognition and a camera

#

is a raspberry pi not ideal for that?

#

so essentially use an online/cloud API

#

gotcha ty

covert aspen
#

Let's say I have a dataframe that consists of 1000 rows. 700 of those rows belong to class A and the rest 300 belong to class B. Is there a function in Pandas that takes in an imbalanced dataframe and randomly removes rows from the class with excess rows such that the final dataframe contains equal number of each class?

untold bloom
#

there is no function to specifically do that

#

but you can identify the major class, get the number of samples to be dropped from it, sample that amount of bad indexes, and drop them from the frame

#

a way

jaunty helm
covert aspen
covert aspen
jaunty helm
#

or actually, if you don't need any randomness you can just .groupby('class').head(100) to get the first 100 of each class (and .tail for the last 100)

sick sonnet
#

what is the absolute fastest embedding model with acceptable performance (must be open source) right now?

agile cobalt
#

the "acceptable performance" will vary greatly based on which task you want to use it for, and the model performance may vary a lot depending on what your data is like

traditional search engine? RAG? Which type of data you're working with?
(QA, text or pdf documents, how structured is it, which language(s))

#

I'm guessing only text, but the way you framed your question it's also ambiguous whenever you want a model that only accept text inputs, be only images, multi-modal

agile cobalt
sick sonnet
#

how does it compare to something more primative such as TfidfVectorizer?

sick sonnet
agile cobalt
#

I'm also not sure about what its output shape would end up as for large dataset?
most embedding models create a vector with few thundred to a few thousand of vectors

narrow tiger
#

i am running mistral on localhost and it is taking a few seconds before responding

agile cobalt
#

unless you're running it on a GPU, that is to be expected

narrow tiger
#

also ollama services is running on my system as soon as my system starts
whenever i call the mistral endPoint does it start mistral each time? bcz i don't see it using too much resources

#

i don't see ollama/mistral using my gpu

#

unless thoose --variation seed versions are ollama??

agile cobalt
#

it should take way more memory than that

narrow tiger
#

thanks i'll try to run it on gpu

#

it isen't even using much of cpu lol

agile cobalt
#

if your GPU does not supports it you could try a smaller model like phi3 but I wouldn't expect amazing performance nor quality from another you can run without a GPU

narrow tiger
#

mistral 7B is running on my CPU i5 5-600k? shouldn't it also be able to run og 1660Ti with 6gb vram i think

lapis sequoia
#

do you ever use training data on a grid search?

spring field
#

bit pytorch specific but what's the difference (if any) between

model.forward(...)
model.forward(...)
...
optimizer.step()

and

with torch.no_grad():
    model.forward(...)
model.forward(...)
...
optimizer.step()
#

that's what I thought, I just don't understand it 😄
like, aren't gradients calculated and added during backprop?

#

oh the gradients of only that layer?

#

for context I was doing A2C RL and the former worked while the latter did whatever it did, lol
also speaking of RL it seemed to me that using too big of a hidden size overfit the score function without actually increasing the total reward

past meteor
spring field
#

I went into the debugger a couple times while trying to figure out why it was doing whatever it did and that never crossed my mind 😁 will do joe_salute

young ledge
#

anyone knows how to make jupyter vs code extension store outputs and variables when i close vs code? kinda annoying having to retrain the model every time i wanna do smt with it after closing vs code
i could save and load the model into a file but jupyter saving outputs wouldve been more convenient

hollow escarp
#

Hi, does anyone know how could i make treshhold for that img which would give the letters really high contrast?'

image.png
hollow escarp
#

I tryed bunch of techniques but even with this img im still having trouble with that OCR

image.png
#

maybe thats because im using easyocr for that task

#

Idk

# 
def preprocess_license_plate_image(img):
    img = cv2.resize(img, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img = cv2.GaussianBlur(img, (5, 5), 0)

    kernel_sharpening = np.array([[0, -1, 0], 
                                  [-1, 5, -1],
                                  [0, -1, 0]])
    img = cv2.filter2D(img, -1, kernel_sharpening)
    img = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, 
                                cv2.THRESH_BINARY, 11, 2)
    
    return img
``` Code used for preporcessing
#

I even achived that preprocessing img

image.png
#

But still having a trouble with correct reading it

toxic mortar
#

Why dont you fine-tune some neural net?

#

Also try tesseract-ocr

#

I had very pleasent exp w it

hollow escarp
#

Ye i know that, but im still conecered about your method becasue Eeasyocr is already using AI model for that

uncut marten
#

does anyone know how yo create a chatbot?

spring field
#

really looking forward to that 4th point lolDog

hollow escarp
#

In good enviorment yes

#

But in field not really

#

i mean i still have some options for different camera setups

spring field
#

anyway, currently I'm doing GAN stuff

hollow escarp
#

Thats S

#

Thats like base img which i preprocess

image.png
spring field
hollow escarp
#

What do you mean by that?

spring field
#

well, a bit in reverse, but yeah

hollow escarp
#

yes but also i know that i didnt setup the camera in best possible way

spring field
hollow escarp
#

I mean i still have a few not tested configurations from which i will definitely start

modest steeple
#

hi i have this pictrue and this is my code:
from PIL import Image
from pytesseract import pytesseract
import enum
import pyautogui
import os

class OS(enum.Enum):
Windows = 1

class Language(enum.Enum):
ENG = 'eng'
ARB = 'arb'

class ImageReader:

def __init__(self, os: OS):
    if os == OS.Windows:
        windows_path = r'C:\Program Files\Tesseract-OCR\tesseract.exe'
        pytesseract.tesseract_cmd = windows_path
        #print("Running on: Windows\n")

def extract_text(self, image_path: str, lang: Language) -> str:
    img = Image.open(image_path)
    extracted_text = pytesseract.image_to_string(img, lang=lang.value).strip()  # Strip the text to remove newlines and spaces
    return extracted_text

if name == 'main':
save_path = r'C:\Users\falcon\Desktop\my_lexis_plus_bot\images1'
file_name = "big_question.png"
save_path1 = os.path.join(save_path,file_name)
left = 1250
top = 400
width = 600
height =42
screenshot = pyautogui.screenshot(region=(left,top,width,height))
screenshot.save(save_path1)
screenshot.show()
ir = ImageReader(OS.Windows)
bigq = ir.extract_text('images1/big_question.png', lang=Language.ENG)
print(bigq)

is there anyway to make my code detect the underscores and write them with the output?
beacuse the output only the sentens without the underscores like this : Nadia is much more than her sister.

big_question.png
narrow tiger
#

use an img reader that supports _

#

how come LLMs are not deterministic if they have predefined weights and they predict what word should come next shouldn't they be deterministic?

serene scaffold
nova matrix
#

I am building a predictive model, consider it to be based on stocks etc. There are columns for search trends which show trends rated from 1 to 100. Could be a very useful variable for the model but most of the data for those search trend columns is missing. Any idea on how I could leverage whatever is given?

winged hornet
#

Is anyone of you experienced in opencv in python

hasty grail
winged hornet
hasty grail
#

This isn't really the place to advertise job openings

#

!rule 9

arctic wedgeBOT
#
Rules

9. Do not offer or ask for paid work of any kind.

hearty token
#

Hi guys, I am attempting to generate vector embeddings for a very structured document. It is divided at four levels, the book, the division, the title and then the content. I have been thinking about how to prepare the material for chunking, especially in a way that maximizes on the rigid structure of the documents. My plan is to divide them by the smallest block, the content, and put a block of text above the content, containing the book, division and title. Every chunk will have this header block and the content. How does this idea sound? One concern I have is the repetition of the header block for divisions that has a lot of titles and content, so it would be the same header block over and over, and I wonder if that will be counterproductive to search.

hollow escarp
#

Anyone knows here how could i quntize my object detection model?

serene scaffold
hollow escarp
#

Sorry for spelling mistake

serene scaffold
#

You might also exclude the header block from each chunk, if it's only there to tell you where chunk boundaries are.

hearty token
buoyant vine
hollow escarp
#

But im still having trouble in doing that

buoyant vine
hollow escarp
#

tried bunch of methods and they didnt speed up My model

buoyant vine
#

they are typically not magic

#

the speedup will depend on hardware

#

typically they only reliably reduce memory overhead, not speed.

hollow escarp
hollow escarp
buoyant vine
#

and what type are you trying to quantize to?

hollow escarp
#

And it didnt work well

buoyant vine
#

I'm not sure if you're going to gain much via float32 -> int8

#

their intermediary results are likely going back to int32 operations

#

I can't see anything wrong with the model itself

#

Looking at the cpu provider code, most of the operations do not work via int8 they get cast internally

#

Well in theory if the data is int8, you can at minimum do 16 operations per AVX2 register on the CPU compared to 8 fp32 operations.
But in arm this is a bit different since NEON & SVE behaves a bit differently

#

But normally smaller int operations = more throughput loosely

hollow escarp
buoyant vine
#

GPUs normally have a lot of specialization around the quantization, so they more commonly see a significant speedup with the quantization

#

But on the CPU you are more at the mercy of the execution provider's willingness to specialize to datatypes

buoyant vine
hollow escarp
buoyant vine
#

Not sure what Arm chip it will be running on so not sure if you can run any of these

hollow escarp
#

Thx, maybe this will give me better resoults

untold wadi
#

Hi there,

I developed a webapp with Flask recently. I'd like to add a scraping task in the background (something I've already dev'd) but things get complicated after :

I have lots of videos, from which I'd like to extract some texts but not necessarily all. I've tested a bit with Google's Vision AI, which works very well, but I end up with a lot of extraneous text. So my question is, what do you think is the best way to clean up the data with OCR tech? I have several solutions in mind, for example training a model to detect only the texts I want (but I don't know where to start with this method), I've also thought about using classic regexes but this solution seems extremely limited and not suitable for me. Can you think of any other viable solutions for me? Any frameworks? Ways to train AI models? etc.

Thanks for taking the time to read already and have a nice day!

buoyant shoal
#

hi, can someone clarify what exactly should i be doing for this question ( i don't particularly understand cuz i'm new to pytorch)

#

Oh yeah the quoted "question 1" is just this:

#

Anyway this is my code for question 1, but idk what to do for q3), could someone help?

mild dirge
#

This should then give you a loss for a single image. Do this multiple times, and and plot the resulting losses. @buoyant shoal

#

Bit unsure about the terminology your teacher uses. The grammar is also a bit poor. I assume with "total loss" (Q1) they mean the "MSE" over 600 images. But that would give the same answer as question 3, so they probably mean "sum of squared error"?

#

Does that make sense @buoyant shoal?

mild dirge
buoyant shoal
# 
# Create a data loader for training data with a batch size of 600
train_dl = torch.utils.data.DataLoader(train_dataset, batch_size=600)```
mild dirge
#

Do you have the entire assignment description?

buoyant shoal
#

The code was like already there for the most part and was just filling in trivial details

#

but iirc they already had batch_size = 100 on the code

#

i just changed it to 600

mild dirge
#

Do you have the original code then?

#

WIthout what you put in it

buoyant shoal
#

oh

#

okay yes it's more or less the same

# 
# Import torch library and the other usual libraries
# Based on code by Naveen on nomidl.com

import torch
import torchvision
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np
from torchvision import transforms```
# 
# Define a data transformation to convert digit images to tensors
transform = transforms.ToTensor()
# Load the MNIST datasets for training and validation
# images are 28x28 pixel images of handwritten digits in a greyscale
train_dataset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
valid_dataset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)

# Create a data loader for training data with a batch size of 100
train_dl = torch.utils.data.DataLoader(train_dataset, batch_size=100)```
#

yeah the rest is all the same actually

#

i moved all the stuffs from the jupyter notebook

mild dirge
#

Hmm alright. There is just some contradicting stuff, like they initally use a batch size of 100, which means they give 100 images, calculate MSE, update model, do this until all images are used, then the epoch is done.

#

But they want to plot the MSE of a batch of 600 for every epoch (for question 1), which is a bit random, Why pick 600 random images and save the MSE of that? why not just use the MSE of all images?

buoyant shoal
#

train_dl = torch.utils.data.DataLoader(train_dataset, batch_size=100)

#

this thing like doesn't even run entirely

mild dirge
#

What do you mean?

buoyant shoal
#

hold on

#

huh okay what

#

it works now but okay basically