data-science-and-ml

[mellow ivy]

Hi!
I have a theoretical question and I’m interested to hear your thoughts about it.

When we speak we create sound waves which could make different objects vibrate.

Would it be possible “in theory” to create a device which could read the vibration of an window for example and translate this into understandable language?

This device is supposed to be from afar

[desert oar]

i think that actually exists. i remember seeing something where some researchers had a video camera pointed at a bag of potato chips, and they could infer with some accuracy what people in the room were saying

i might be mis-remembering

https://news.mit.edu/2014/algorithm-recovers-speech-from-vibrations-0804

[mellow ivy]

Ah! Cool! Thank you! 🙏

[desert oar]

https://youtu.be/FKXOucXB4a8?t=122 that's an impressive result. i'm sure with improved hardware and algorithms nowadays they could do even better

[stark bough]

hi people

can i ask for help in ETL issues in AWS Glue?

with pyspark?

[desert oar]

you can. i saw your question in #databases , but i think the (very few) spark users here are more likely to see it in this channel

however your question is very specific and i don't know if anyone here has experience with what you're asking about

i've certainly never had to process xml with spark

[stark bough]

oh well understood.

[desert oar]

it would help if you posted your code, then maybe someone can take a look

!paste

[arctic wedge]

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[main fox]

(I'm on mobile)
Comparing SQL queries to Pandas equivalents. Although the pandas equivalent could probably be better.

[royal crest]

i want to help but my neck pain is too immense

[tropic stream]

Does anyone know how Pretext tasks apply to Downstream task in Self-Supervised Learning?

[wanton sleet]

how can we make api for tensorflow model?

can anyone provide me some links about this?

[tropic stream]

do you have your AI model trained?

[wanton sleet]

yes already

[tropic stream]

then just export a pickle file and wrap it using any Python API framework such as FastAPI 🤔

[next lance]

I am also leaning but is there any way to not make sooo many folders and setups for Tenserflow Image detection API

[wanton sleet]

okay thanks

[next lance]

I wanna learn like something basic, I think I mostly spoon feeding myself

[tropic stream]

This series with simple explanations and hand-holding steps might help you discover more about hands-on experience with Tensorflow

1. https://willogy.substack.com/p/tensorflow-insights-part-1-image
2. https://willogy.substack.com/p/tensorflow-insights-part-2-basic
3. https://willogy.substack.com/p/tensorflow-insights-part-3-visualizations
   Hope this helps

[next lance]

Cool, I think I will be able to learn why we make so many folders and how to detect objects,Thanks

[velvet thorn]

screenshot + rotated

that’s the ultimate sin

[main fox]

😔 guess I'll burn

[lapis sequoia]

The time complexity is also a factor. Not just the result. Also I assume you are doing on some small data. You should check the results of time too on big data before the conclusion.

[main fox]

On big data, SQL will outperform pandas

[lapis sequoia]

Exactly.

Equivalency is more of isomorphic word than homomorphic.

[main fox]

There wasn't really a point with the image when I sent it, other than "look how these compare"
But thinking on it now, I'd assume a big part of preprocessing would be done in SQL

When working with big data

[lapis sequoia]

Tho I'm pretty sure there would be ways to create good indexes in pandas to out perform what we simply do of course.

[scarlet cairn]

i have a question

what if i had a website and i wanted to make a button that'll send them an email, do I need to learn ai for that?

[polar widget]

Hi guys, does anyone know anything about Random Forests?
I am trying to construct a subbagging random forest. Can someone look over what I have built so far?

I want to form the forest over binom(n,s) possible s-element subsets of X. The subsets are to be created by drawing without laying back and no subset is to be used twice, i.e. I can form at most a forest of binom(n,s) trees.
My idea is, I form all possible index sets first. Then I shuffle. And take the first N pieces and build a tree for each, using the whole index set. With N I mean the number of trees in the forest ( N <= binom(n,s) )

import numpy as np
import random
from numpy import linalg as LA
from sklearn.ensemble import RandomForestRegressor as RF
from sklearn.tree import export_graphviz
import pydot
import math
import itertools
random.seed(10)
np.random.seed(10)

n = 5
p = 3
s = 3
# Max Combinations
l = math.comb(n,s) 
print('Anzahl der maximalen Möglichen Teilmengen bei Ziehen ohne Zurücklegen beträgt {}.'.format(l))


#Erstelle eine Liste mit allen Kombinationen bei s-maligen Ziehen ohen Zurücklegen aus 1,...,n
C_N = list(itertools.combinations(range(n), s))
random.shuffle(C_N)
C_N = np.array(C_N)

################################
#Regression with Y = f(X)  + \epsiolon
#with f(x) = norm(x) = || x ||
X = np.random.uniform(0, 10, size=(n, p))
Y = np.array([ LA.norm(X[i]) + random.normalvariate(0,1) for i in range(n) ])

X_0 = np.random.uniform(0, 10, size=(1, p))
Y_0 = LA.norm(X_0)
###############################
#Number of Trees used
N = round(n/2)

M = 1
mtry = 2
nodesize = 2

for i in range(N):
    rf = RF(n_estimators=M, random_state=10, max_samples=s, max_features=mtry,
            min_samples_split=nodesize + 1)  #

    rf.fit(X[C_N[i]],Y[C_N[i]])
    RandomTrees[i] = rf.predict(X_0)

print(RandomTrees)
print(RandomTrees.mean(),Y_0)

[prisma mulch]

what different types of models are there apart from sequential in keras?

that was exceedingly fast to delete that spam

[lapis sequoia]

could someone guide me how to make prediction of audio spectrogram from already trained model?

[prisma mulch]

https://dolby.io/blog/beginners-guide-to-visualizing-audio-as-a-spectogram-in-python/

[lapis sequoia]

this page don't have a code

to predict i am new to this

i copied my code from here

https://www.kdnuggets.com/2020/02/audio-data-analysis-deep-learning-python-part-1.html

i just want the predict part where i fit audio into an array and put it in model to predict

[prisma mulch]

I thought you wanted to visualize

look into wavenet

[pallid badge]

Hi, I am trying to learn xarray. I have not understood what are coordinates. There are dimensions too. I wish to read many hdf5 files, extract the numpy arrays (3D), and store all of them in an xarray dataarray probably. I used xr.open_dataarray to open hdf5 files, but I am not sure if they are really open, for example. I don't find anybody I can chat with this about. Hope, maybe here somebody has some knowledge. Thank you.

[inner anchor]

is ML just about csv files?

[lapis sequoia]

what does this mean?

https://nothing-to-see-he.re/5d_O1D9F2.png

https://nothing-to-see-he.re/5d_NXQdSt.png

[lapis sequoia]

this

[hasty grail]

That didn't work well with Flask, the table created in memory before running the application "doesn't exist" when handling user requests (I suspect it's because of threading issues)

Ended up using temporary files

import os
from pathlib import Path
import uuid
from weakref import finalize

from sqlalchemy import create_engine

def _create_workspace_engine() -> Engine:
    workspace_dir = Path(__file__).parent / 'workspace'
    workspace_dir.mkdir(exist_ok=True)

    filepath = workspace_dir / str(uuid.uuid4())
    while filepath.exists():
        # Avoid overwriting existing file
        filepath = workspace_dir / str(uuid.uuid4())

    engine = create_engine(f'sqlite+pysqlite:///{filepath}')

    # Free up storage space automatically
    finalize(engine, os.remove, filepath)

    return engine

[swift oxide]

Hi

So how can I use SQL and Pandas simultanously

[serene scaffold]

specifically, are you asking how you can use pandas to connect to an SQL database?

!docs pandas.read_sql_query

[arctic wedge]

pandas.read\_sql\_query

pandas.read_sql_query(sql, con, index_col=None, coerce_float=True, params=None, parse_dates=None, chunksize=None, dtype=None)```
Read SQL query into a DataFrame.

Returns a DataFrame corresponding to the result set of the query string. Optionally provide an index\_col parameter to use one of the columns as the index, otherwise default integer index will be used.

[serene scaffold]

there's a few other functions for reading SQL stuff.

[hollow ember]

@serene scaffold help please

[serene scaffold]

do you understand the error message?

[hollow ember]

no

[serene scaffold]

the index argument has to be a list or something

it's not the name of the index column

[hollow ember]

so instead of an array i need to make a list?

[serene scaffold]

right now you have index='Price', presumably because you want the name of the index column to be "Price", yes?

[hollow ember]

Yes

[serene scaffold]

but that's not what the index= argument is for.

[hollow ember]

oh

[serene scaffold]

you pass something that can actually be used as an index

[serene scaffold]

did that work?

[hollow ember]

yeah

[serene scaffold]

[hollow ember]

@serene scaffold

the whole thing

index with weekdays worked tho

but the below index still gives out an error

[serene scaffold]

I can't know what you mean by "it gives out an error" without seeing the whole error message.

[weak tiger]

why nobody answered me?

[hollow ember]

@serene scaffold https://paste.pythondiscord.com/atimarusoc.py

[serene scaffold]

sales=pd.DataFrame(prices.reshape(1,3), index="Price", columns=["Almond","Peanut","Cashew"])
We already talked about why you can't pass a string for the index= argument.

[hollow ember]

Bruh i suck, i forgot to add [] in index="Price"

it worked now

[serene scaffold]

[hollow ember]

thank you @serene scaffold

[lapis sequoia]

Hi... Everyone...

Is anyone conversant with skfuzzy... I've been having an error for days...

[slim spear]

Can somebody help me understand dtree.predict() ? I can't find the docs for it.

Nevermind.

[quasi sparrow]

If python is used for data science: where do R programming languages come in handy?

[safe birch]

Can someone help to look through all my 8 puzzle problem program. I am struggling with how to use # to represent 0. I can run my programme but I can't use # to represent 0.

[slow vigil]

Anyone know how to scrape data from a website when all the html is inside a Node webapp?

[wise pelican]

So I'm taking creating exponentially weight moving window object from pandas for a given series of data
EWM only supports mean, variance, std dev, cov, and corr functions, but I'd like to aggregate my own functions on to it.
Problem is that ExponentialMovingWindow objects don't support this
Is there a way I could either apply other functions to an ExponentialMovingWindow series or do something like calculate the EWM myself into a series?

[tidal bough]

Looking at the source of, say, the variance:
https://github.com/pandas-dev/pandas/blob/v1.3.4/pandas/core/window/ewm.py#L498-L531
they end up doing ._apply. So you can maybe apply your own function, though you'd need to be careful to make your function have the same form as theirs

It may be easier to calculate your own EWM

[velvet thorn]

wow

I literally just looked at the source too

and was about to say the same thing

I wonder why it's not part of public aPI

[tidal bough]

it might just be because _apply takes some pretty scary function

[wise pelican]

I was just going through the source code but you beat me to the good stuff, thank you!

[distant trout]

Anyone know newton algorithm with adapting step size?

[median idol]

Hey guys, maybe there is anyone that is good at data science as I wanted to ask help about studying it. could anyone please advise me is my approach is good for learning skills to become data scientist. As I’m in the process of learning. I thought of doing learn- practice this particular topic - move on to the next topic. And repeat in that pattern until I come to learn scikit ( Right now I’m practicing pandas, seaborn ) . And only after I would start looking at others good and doing my own projects connecting all the knowledge I got. Do you think it is fine to study like this ? Or maybe is it better to start already looking through others code? ( but I think that I will not understand much only knowing Python and vital libraries: pandas, seaborn, numpy )

[lapis sequoia]

There is no exact thumb rule. You can look at others' code if it helps. But as i said there is no thumb rule. Going through one library at a time and being efficient in them does seem like a good approach. I'd say you can stay on this path and if you want you can try other ways too! Whatever floats your boat is best way for learning.

[distant trout]

Best idea is to get somehow first job and work with ppl better than you.

[stoic musk]

Hi all, I'm running my first solo Kaggle competition and am running into a data imbalance issue.

The challenge is a binary classification problem, with 13 features and 299 examples, with the following distribution:

0 203
1 96
Name: DEATH_EVENT, dtype: int64

So far, I've:

1. resampled the positive cases (doubled the positive cases to balance the data set)
2. run 3 models on the resampled data.

The best performer so far has an accuracy of ~84%

How else would you balance the data to garner more accurate results?

[wheat orbit]

hi! anyone with experience with numba's parallel mode here?
I'm basically applying a map over a large numpy array (~2 million rows) which looks roughly like this:

for i in prange(arr.shape[0]):
    x = compute some stuff with arr[i]
    arr[i] = x

arr is a matrix but I am mapping over the rows
given the computations in each iteration are rather expensive I'd like to parallelize it (thus prange)
My question is: when I write into arr in a parallel loop it is fine for the parallel code to reference the shared array since they write to different sections. Does numba do this or does it make copies of the array for each parallel thread?

[prisma mulch]

may I bump?

[charred creek]

Does anyone here have experience working with SB3 (stable-baselines3) A2C model? I have a question about its reward range, and how it works if someone can help please. Thank you! 🙂

[lapis sequoia]

how to import a kagale data set?

into tensorflow

[bitter harbor]

im having a bit of trouble trying to use requests, I'm getting the request back but the api says it returns a json list but I'm only getting the html and im not sure why that is/how to parse it

nvm I just was using the wrong url format

[tender hearth]

You download the dataset

If it's a common dataset there's probably already a utility class in TensorFlow to interface with the dataset

You just need to download it

If it's not a common dataset you need to write your own class to read and parse the dataset

[lapis sequoia]

oo i see

so like how do i find out if it has utility class?

i mean it was on the home page i would assume it is

[tender hearth]

can you share the link?

These are links to all utility classes that TensorFlow has

https://www.tensorflow.org/datasets/catalog/overview#all_datasets

[lapis sequoia]

https://www.kaggle.com/fedesoriano/heart-failure-prediction

idk what i am doing tbh

i just wanted to build a mode after i learned how to make a linear regression model

[tender hearth]

Ah alright

Well you download the .csv here

Then you use a library like pandas to load it into a nice object that has useful methods and can easily interact with

[robust charm]

Hi quick question. If I make a model using stable_baseline3 can I use Tensor flow Lite so that I can use the model on my microcontroller?

[grave frost]

are you referring to functional API?

[hollow ember]

Last time i used img it worked but now it doesn't.

[serene scaffold]

it looks to me like it's trying to parse and execute that as Python

[hollow ember]

yeah

[serene scaffold]

that's definitely HTML or something.

Python only uses < as an infix operator, so it can't be the start of an expression

[hollow ember]

yeah

[royal crest]

can't have img src as a variable name either

[hollow ember]

oh i got it the cell needs to be in MARKDOWN option

thx

[median fulcrum]

Hello, I had already done a facial recognition of people even without classifying who the person is. I have a project that requires me to have a training database and then the algorithm tells me who that object and/or person is. I would use Pillow and open cv. Any articles, docs, or materials that talk about this? I searched, but I end up finding more about facial recognition in video etc.

[late olive]

could someone please help me with a datascience ish problem in #help-chocolate

[lapis sequoia]

I'm close

[lapis sequoia]

Wrong channel bruh

[prisma mulch]

what are the default arguments of all the layers?

[stark bough]

Hi guys. Here is the Glue ETL job code that is causing headaches to me

That line takes almost 4 minutes to execute for just one register
this job tries to struct XML texts
these texts are stored un pyspark dataframe column. Very long texts
and i need to loop that column to pass texts to extract functions. Tried with collect() and ToLocalIterator() and now with ToPandas()
have the same result. Really need your help guys :c

[wicked grove]

@desert oarim really stuck again, i tried logistic regression too and i got the accuracy as 78%
I think my code follows the tutorial, but the accuracy never hits 80

[vague relic]

Does anyone know how to extract details like street name, House name, pincode etc. from a string that represents full address?

[somber prism]

anyone know how to implement rating system like kaggle does

i need an algorithm

[eternal niche]

Heya Guys!

Nice to join you!

I am an Electrical and Electronic Engineering student

And just started a masters In machine learning!

So I’m very new to this

I’m starting to use pytorch

And for now I’m learning basic linear regression

Using neural networks

But I am struggling to understand what this line is intentes for:

(1 sec)

[somber prism]

damn bro dont go straight to neural netwowrks

[eternal niche]

What is this LeakyReLU exactly used for?

[eternal niche]

Hahaha thanks a lot, but it’s one of the first modules on the masters hahaha

And I’m just lost hahaha

I tried searching online, and play around to see how the model changes, but I haven’t got a clue

[vast yacht]

hi guys, any recommendation on Data Science/Machine Learning books?

[somber prism]

learn ml for stanford by andrew ng

best one to get into ml/dl/ds

[somber prism]

its an activation function, there are lot of activation function each one have their pros and cons

can someone tell me how to implement rating system like kaggle does ?

[eternal niche]

Thank you very much!!!

[topaz grove]

Anyone can help me with this

?

Original image is

I wanna know where this line will end

The red line is what i wanted

but i dont know how to do it

[topaz grove]

Still cant get location of it

[lapis sequoia]

you trynna make a hack for 8 ball pool xD

[burnt knot]

So I haven't been able to figure out how to successfully build my voice cloning model
I'm following the steps
But I think it comes down to whatever kind of dataset I downloaded for building the model

I mean, I have an English dataset and a Japanese dataset
But when I encode and synthesise the audio for them, no matter which dataset I decide to use (I have switched them out) I can't get to move on with the process

[lime ocean]

Does anybody know what I am doing wrong here?

model = make_pipeline(TfidfVectorizer(), MultinomialNB())
model.fit(train['Consumer_complaint'], train['Product'], tfidfvectorizer__ngram_range=(1,1), multinomialnb__sample_weight=sample)

it works just fine until I add the arg tfidfvectorizer__ngram_range=(1,1)

then I get
TypeError: fit_transform() got an unexpected keyword argument 'ngram_range'

as far as I can tell, TfidfVectorizer should accept an ngrams arg

it says it does in the doc :/

so I think I must be doing something wrong, doubt it's sklearn's fault

https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html

[median fulcrum]

|___| <--- 👆 --->

[uneven stirrup]

#help-popcorn anyone know anything about this

[zealous burrow]

Hi everyone
I am beginner of ML
I don't understand should I start learn ML with scikit learn or google cloud platform ?
What the difference between them?

[neon imp]

One is a platform by Google, basically a whole bunch of different technologies that do different things. SciKit is a python library. You should follow a course to get your head in the game and learn some basics.

[frail oxide]

I think RealPython did a few examples. I dont know anything about ML, but I remember seeing the article. Maybe check this? https://realpython.com/tutorials/machine-learning/

[olive shore]

Anyone good with ai related to linear regression and stock market prwdiction? I need help

[wicked grove]

i was getting a similar error when i used tfidf,try passing analyzer = lambda x:x in TfidfVectorizer()

[lime ocean]

oh man, I just realized what I was doing wrong 😅

[lime ocean]

ngram_range is an argument to TfidfVectorizer's constructor, not model.fit()

[lime ocean]

thanks for the help, wouldn't have figured it out without you :)

[main fox]

Generally, linear regression and ARIMA models perform poorly for this. Predicting the stock market in this way is almost impossible. There is a python package by Stefanie Molin you can check out that pulls stock market data and performs forecasting with those two models if you want to check it out. It's a GitHub repo.

[austere swift]

its ReLU but it allows values below 0, although shrunk by a small factor (used to fix the diminishing gradients that ReLU can sometimes cause)

in case you didnt know what ReLU is, it's pretty much just max(0, x)

so if its less than 0, it returns 0, if its greater than 0 then it leaves it untouched

LeakyReLU would be like 0.01*x if x < 0 else x

(0.01 is the negative slope parameter, which is adjustable but defaults to 0.01)

[zealous burrow]

@neon imp thanks for replying
Because the Google cloud got offer ML course to someone with zero knowledge about ML while
Scikit learn is always said to be the first library for ML beginner to learn
So its really confusing me of which should i choose

[zealous burrow]

Tks @frail oxide
For sharing a great tutorials

[dim herald]

hi guys is there anybody who understands something in docker (especially in airflow running with docker)

[median fulcrum]

|___| <--- 👆 --->

[peak halo]

2021-10-30 17:42:26.601289: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-10-30 17:42:26.601390: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2021-10-30 17:42:26.864031: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2021-10-30 17:42:26.864189: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Epoch 1/10
2021-10-30 17:42:26.949280: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.

what do i do??

im using an M1 mac

[void sail]

is there something I can do to make the training smoother? (pretrained resnet18)

or is this to be expected, the volatile curve

[edgy brook]

Hey guys, do you need to find the coefficients of Stochastic Gradient Descent in order to evaluate the predictive model? Cus for knn, I think it's not needed

[bronze skiff]

?? coeffs of sgd? you mean like the learning rate? and knn doesn't use sgd... or any iterative process to train

[velvet thorn]

well, I suppose technically if you're building some sort of data structure to avoid bruteforce search during inference, that might be considered iterative 🥴

[bronze skiff]

i'm guessing that at the point you gotta do that, you're already too high dimension for knn

gotta go for that sweet sweet LSH

[velvet thorn]

can you elaborate on that reasoning

p sure sklearn uses some sort of tree by default? I think it's a ball tree

as long as it has more than 15 (I think) samples?

(unrelated to dimension, which I understand to be feature count)

[bronze skiff]

yes, you're right i was confusing this with sometime else

[velvet thorn]

I didn't know this was a thing

TIL

that's p cool

is it commonly used?

[bronze skiff]

yes, especially since in high dimensions knn becomes useless

[velvet thorn]

I mean, in general

like outside of KNN

[bronze skiff]

no idea

only ever used it for that

though the wider scope of "probabilistic data structures" is useful for any large data analysis

[wintry ether]

hello can any1 help me make a ai minecraft bot?

which u can give commands and it does it

[median fulcrum]

|___| <--- 👆 --->

[shut trail]

sort of like a sentient Alexa for minecraft ?

[kindred whale]

hi,
how i can turn off function in jupyter notebook, when i running my code block new code block making auto, how to turn it off?

[serene scaffold]

You could comment it out?

[kindred whale]

sec

[slim bone]

Is there a website that gathers quality ML resources into it like Full-stack Python?

[serene scaffold]

what do you mean by "full-stack python"? "full stack" usually refers to a web developer who can do front and back end development. it doesn't really have a machine learning engineer analogue.

[slim bone]

Oh I meant the website "Full-stack Python". It's a site with a long list of useful resources that cover nearly all of the aspects of back-frontend web development in Python
I was wondering if theres something similar to the AI and Machine Learning field.
Should've been clearer, my apologies @serene scaffold

[serene scaffold]

For future reference, editing a message and adding a mention doesn't cause a ping.

But no problem. And I'm not really familiar with one. Maybe try Kaggle?

[slim bone]

Never heard of Kaggle, I'll look into it

[serene scaffold]

do you have any prior experience with AI?

[slim bone]

Not at all

[serene scaffold]

keep in mind that there's going to be a lot of math and the "rate of reward" probably isn't going to be as favorable as with other areas of programming. you'll need to become comfortable with probability, statistics, combinatorics, and linear algebra.

[kindred whale]

stelercus

how i can make it

lol

:(

[slim bone]

@serene scaffold Good thing you told me. Is there any place where I can strengthen my mathematical knowledge?

[serene scaffold]

I dunno, khan academy? I did it the old fashioned way (four year degree)

[slim bone]

Software engineering or something else?@serene scaffold

[serene scaffold]

computer science/data science

[kindred whale]

im new in data science and learn numpy now

[slim bone]

I see, and you had experience with AI in Uni/College?

[serene scaffold]

one should not seek to "learn numpy" but instead do things where they could accomplish what they're trying to do more effectively with numpy.

[serene scaffold]

ye

[slim bone]

Ah, I see. And you dont know about resources that teach ML from scratch?
Also, are all of the topics you mentioned earlier(probability, statistics...) cover the field or AI? Or are there other topics I should focus on as well?

[kindred whale]

i have one question

[serene scaffold]

so, one of the most common things one does in AI is classification. which is where you build a program that can take something and say what group it belongs to

[kindred whale]

why when i makin

a = np.empty((5, 2), dtype='int16')

I have result

array([[123,   0],
       [  2,   0],
       [  3,   0],
       [  4,   0],
       [  5,   0]], dtype=int16)

```,
why 123?

[serene scaffold]

for example, you might have a program that tries to guess if someone is a man or a woman based on their height and weight. statistically, men are taller and heavier

so the program might learn that if someone is above a certain height, the probability of them being a man is, say, 60%.

[serene scaffold]

do you know how numbers are stored inside a computer? np.empty just allocates an array of the given shape, and the numbers you get just depend on the current state of whichever bits Python decides will be part of the array.

[median fulcrum]

|___| <--- 👆 --->

[serene scaffold]

as opposed to np.zeros, which will flip all the bits to 0 for you.

!e import numpy as np; print(np.empty((5, 2), dtype='int16'))

[arctic wedge]

@serene scaffold :white_check_mark: Your eval job has completed with return code 0.

001 | [[     0      0]
002 |  [     0      0]
003 |  [     0      0]
004 |  [     0      0]
005 |  [-17360  -4320]]

[wintry ether]

yh

[lapis sequoia]

is this where i can ask questions ab torch?

[slim bone]

@serene scaffold I see, so focus on classification. Any other topics?

[serene scaffold]

I was just giving an example of how statistics is used in AI

[slim bone]

Ah, I see

[serene scaffold]

there was probably a better one I could have thought of

[slim bone]

No its okay I got the point I believe

[serene scaffold]

but yes, if you make your first goal "make a classifier for some dataset on Kaggle", that should keep you positively occupied for a while.

[slim bone]

@serene scaffold Will Kaggle teach me how to do that though?

[serene scaffold]

I'm not sure.

I must disappear for a while

[slim bone]

Okay, you've been very helpful

[lapis sequoia]

I'm tryna make an ml model w torch, but torch doesnt notice my gpu

[slim bone]

Thank you very much

[lapis sequoia]

only thing is that i downloaded the drivers for my gpu, i downloaded cuda, and i downloaded torch w cuda

so youd expect it to work, but torch still doesnt pick up a gpu

[serene scaffold]

Show the command you used to install torch

Also what os

And what gpu

[lapis sequoia]

pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
this is the command i used
I'm using Windows 10
GTX 1050 Ti on a laptop

i pulled the command straight from the pytorch website

[wicked grove]

🙈🙈no problem:)
I just read what a constructor is,should i start learning oops to know more about classes,constructors?

[serene scaffold]

okay, and then what code did you run that communicated to you that torch hasn't noticed your gpu?

and what was the result?

[serene scaffold]

OOP is more than just "having classes" and includes some software design concepts that Python doesn't really encourage.

[wicked grove]

Ohh okayy!!

Could you please check my final code
https://paste.pythondiscord.com/ocosulaxef.py
I got this accuracy from it

           0       0.77      0.77      0.77     39752
           1       0.77      0.77      0.77     40248

    accuracy                           0.77     80000
   macro avg       0.77      0.77      0.77     80000
weighted avg       0.77      0.77      0.77     80000

I tried to match the code to the tutorial i was following and this is the max i can get

Could you please tell me if this is okay

[lapis sequoia]

im using this module detecto that uses torch under the hood:https://detecto.readthedocs.io/en/latest/usage/quickstart.html

when i go to actually train the model it tells me it's only using cpu, lemme run it rq to get the message

https://paste.pythondiscord.com/yabokusave.py here's the code

[serene scaffold]

@lapis sequoia for one thing, do python -c "import torch; print(torch.cuda.is_available())" and show the result.

[lapis sequoia]

ya thats the problem tho

[serene scaffold]

what does it display

[lapis sequoia]

even after installing torch w cuda it shows up as false

[serene scaffold]

try nvidia-smi and copy/paste the whole thing into the paste bin

!paste

[arctic wedge]

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[lapis sequoia]

nvidia-smi?

[serene scaffold]

yes, it's a bash command.

also works in powershell

[serene scaffold]

in the future, please always copy and paste everything as text; I won't read screenshots.

except this one

[lapis sequoia]

kk

https://paste.pythondiscord.com/inuxazijoq.rb

[serene scaffold]

what happens if you do pip install torch==1.10.0+cu113, without the -f part?

the cu113 part means "cuda 11.3", but you have 11.5.

granted, so does +cu113, but if you don't use the -f part, that might force it to compile locally instead of using a precompiled version

[lapis sequoia]

ERROR: Could not find a version that satisfies the requirement torch==1.10.0+cu113 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0) ERROR: No matching distribution found for torch==1.10.0+cu113

[serene scaffold]

okay, let me see

[lapis sequoia]

also

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

it should be in pypi tho

[serene scaffold]

try pip install --pre torch -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html

(that's what worked for this person five days ago: https://discuss.pytorch.org/t/error-installing-with-python-3-8-and-cuda-11-5/134914/6)

[lapis sequoia]

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Looking in links: https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html Requirement already satisfied: torch in c:\users\ashib\anaconda3\lib\site-packages (1.10.0) Requirement already satisfied: typing_extensions in c:\users\ashib\anaconda3\lib\site-packages (from torch) (3.7.4.3)

lemme run rq

[serene scaffold]

oh, you're using anaconda?

[lapis sequoia]

still false

oh what no

[serene scaffold]

so why does it say Requirement already satisfied: torch in c:\users\ashib\anaconda3\lib\site-packages (1.10.0)

[lapis sequoia]

i installed conda to see if having a diff interpreter would work

um

thats a very good question

[serene scaffold]

try which pip

it might not work depending on which terminal you're using

[lapis sequoia]

says which isnt a cmdlet

[serene scaffold]

let me find the equivalent command

[lapis sequoia]

its going thru conda when i run which pip in git bash

[serene scaffold]

okay, try conda install cudatoolkit=11.3

[lapis sequoia]

/anaconda3/Scripts/pip

kk

[serene scaffold]

and then do conda install torch again

I don't know how to run python programs with conda, though.

[lapis sequoia]

im using vsc to run my code

[serene scaffold]

also funny thing, I haven't eaten today. I think I need to do that.

[lapis sequoia]

do taht then

im here all day

[lapis sequoia]

!paste

[mystic prairie]

I wanted to make an AI that could generate a certain type of images given a dataset, I have no background knowledge where should I start learning things

[lapis sequoia]

mit has a really great online lecture series ab ml

ml being machine learning

i think they also go through code

that's where im starting

[lapis sequoia]

could some one help me at #help-avocado message

[lapis sequoia]

@serene scaffold its still downloading stuff

or no, its examining conflicts

warnings.filterwarnings('ignore')

cmap = plt.get_cmap('inferno')
plt.figure(figsize=(8,8))
genres = 'blues classical country disco hiphop jazz metal pop reggae rock'.split()


for g in genres:
    pathlib.Path(f'img_data/{g}').mkdir(parents=True, exist_ok=True)
    for filename in os.listdir(f'C:\\Users\\jimmy\\Downloads\\genres\\{g}'):
        songname = f'C:\\Users\\jimmy\\Downloads\\genres\\{g}\\{filename}'
        y, sr = librosa.load(songname, mono=True, duration=5)
        plt.specgram(y, NFFT=2048, Fs=2, Fc=0, noverlap=128, cmap=cmap, sides='default', mode='default', scale='dB')
        plt.axis('off')
        plt.savefig(f'img_data/{g}/{filename[:-3].replace(".", "")}.png')
        plt.clf()

header = 'filename chroma_stft rmse spectral_centroid spectral_bandwidth rolloff zero_crossing_rate'
for i in range(1, 21):
    header += f' mfcc{i}'
header += ' label'
header = header.split()

file = open('dataset.csv', 'w', newline='')
with file:
    writer = csv.writer(file)
    writer.writerow(header)
genres = 'blues classical country disco hiphop jazz metal pop reggae rock'.split()```

for g in genres:
    for filename in os.listdir(f'C:\\Users\\jimmy\\Downloads\\genres\\{g}'):
        songname = f'C:\\Users\\jimmy\\Downloads\\genres\\{g}\\{filename}'
        y, sr = librosa.load(songname, mono=True, duration=30)
        rmse = librosa.feature.rms(y=y)
        chroma_stft = librosa.feature.chroma_stft(y=y, sr=sr)
        spec_cent = librosa.feature.spectral_centroid(y=y, sr=sr)
        spec_bw = librosa.feature.spectral_bandwidth(y=y, sr=sr)
        rolloff = librosa.feature.spectral_rolloff(y=y, sr=sr)
        zcr = librosa.feature.zero_crossing_rate(y)
        mfcc = librosa.feature.mfcc(y=y, sr=sr)
        to_append = f'{filename} {np.mean(chroma_stft)} {np.mean(rmse)} {np.mean(spec_cent)} {np.mean(spec_bw)} {np.mean(rolloff)} {np.mean(zcr)}'
        for e in mfcc:
            to_append += f' {np.mean(e)}'
        to_append += f' {g}'
        file = open('dataset.csv', 'a', newline='')
        with file:
            writer = csv.writer(file)
            writer.writerow(to_append.split())

data = pd.read_csv('dataset.csv')
data.head()# Dropping unneccesary columns
data = data.drop(['filename'],axis=1)#Encoding the Labels
genre_list = data.iloc[:, -1]
encoder = LabelEncoder()
y = encoder.fit_transform(genre_list)#Scaling the Feature columns
scaler = StandardScaler()
X = scaler.fit_transform(np.array(data.iloc[:, :-1], dtype = float))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)#Dividing data into training and Testing set

model = Sequential()
model.add(layers.Dense(256, activation='relu', input_shape=(X_train.shape[1],)))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])```

classifier = model.fit(X_train,
                    y_train,
                    epochs=100,
                    batch_size=128)
score = model.evaluate(X_test, y_test, verbose=0)
print(score)
model = keras.models.load_model('audio classification.h5')```

well ihave this code which convert a audio file into spectogram
i just want a predict function
in which function i give audio file and it predicts which class it belongs to

!paste

[median fulcrum]

|___| <--- 👆 --->

[arctic wedge]

:incoming_envelope: :ok_hand: applied mute to @narrow basin until <t:1635635061:f> (9 minutes and 59 seconds) (reason: duplicates rule: sent 4 duplicated messages in 10s).

[empty aurora]

let's say I have two columns, entry and exit, and I want to count how many exits happen within 30 minutes of each entry, is there a way to do it without apply?

[velvet thorn]

((df['exit'] - df['entry']) <= timedelta(minutes=30)).sum()

[empty aurora]

thanks, this works if exit and entry are same row, i want to count how many entries happen within 30 minutes of each entry

example: if row 1 is entry 8 am, i want to count how many entries happen within 8 and 830 am

def next_closest_dep_calc(row):
    nxt_dep = (tstflights['STD_UTC'][
        (tstflights.STD_UTC <= row.STA_UTC + pd.Timedelta("600 minutes"))
        & (tstflights.STD_UTC > row.STA_UTC)].min())
    print(f"{x.STA_UTC} next dep {nxt_dep} in {nxt_dep - x.STA_UTC} minutes")
    return 0
tstflights['nuevo'] = tstflights.apply(count_deps_52mins, axis=1)

[velvet thorn]

oh.

[empty aurora]

this is what I am doing right now, with an apply, and of course here i am finding the closest STD_UTC to each rows STA_UTC

[velvet thorn]

that's more complicated

[thin palm]

Hi, can someone explain weights to me in Deep Neural Network?

Because when I Google this there is no simple solution

[velvet thorn]

not sure if it can be done

faster than quadratically...?

[empty aurora]

i've no idea either, one way i was thinking was using joins, which i have done in the past its like a super application of branch and cut something like that lol

[velvet thorn]

like cross joins?

it's not hard

[empty aurora]

yeah. and then count and go back to the first df

[velvet thorn]

you could use like np.subtract.outer

[empty aurora]

but now you counted what joined

mmm

thats nice, ill explore that numpy is always super fast

thanks

[velvet thorn]

which would make the most sense to me

there might be a faster than quadratic algorithm

not sure

[undone mirage]

hello i want to learn about machine learning, where should i start?

[simple ivy]

hey yall! question- when implementing object detection with custom trained models, does the process always require that you manually label images in your dataset?

[lapis sequoia]

i just recently got into this, but usually, ya

unless u can find a dataset online

theres a ton of datasets online

or ya know

u can always pay someone to label for ya

or just get like a 5 yr old a chocolate bar for a few hours worth of work

child labor for the win

[simple ivy]

hahaha

wait but is that also the case for datasets with over 100,000+ rows of data?

that doesn't seem... possible to label each and every single image

[lapis sequoia]

ya im pretty sure theyre usually hand labeled by a shit ton of people or theres already a pretrained model churning out datasets

[rugged oyster]

Hi, We want to build python 3 (latest release) on windows for a xeon machine we are testing against nuclear and solar radiation. What build syntax would I use that would include PGO as well as remove source files and leave only the required files/folders (../lib, ../include, etc...) on a win7pro machine? (xeon) The machine will be doing a lot of mobile device testing. The build needs to be secure and geared towards working with devices over usb-c.

[vague relic]

How do AI engineers know what machine learning model to use for a particular task? For example, if it is a whether prediction task and you've been given some variables like date, latitude, longitude, elevation, etc. How do you know what model to choose?

[rugged oyster]

Optimally a added install switch to have a shadow copy auto generate would be the best route in addition to that. Thank you all so much!

[quiet vault]

For this kind of problem, a neural network will probably be the best option because there are multiple variables to learn

Choosing the model really depends on how difficult the problem is to predict for, the time that you have to develop the model and what kind of problem it is

For example, you have a time series dataset and it is a regression problem. If the problem is not very complicated, and you don’t have much time you should choose ARIMA or SARIMA. If you have a lot of time and the problem is difficult, you could chose deep learning such as lstm

Nearly every computer vision task these days should use a CNN. There isn’t really any exceptions on that one

[quiet vault]

Yeah ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an anual computer vision competition and it has around 1 million images. I’m pretty sure they are all done by hand

[quiet vault]

I would start with a 6 hour long tutorial video from TechWithTim. i am pretty sure the video is called “Deep learning for beginners”. It explains the concept and application of tensor flow on multiple different types of models which is really cool. From there, you can look at different resources to expand your knowledge in specific fields

[edgy brook]

Hmm, I think so? I actually don't know since we only covered this in a lecture before we're handed an assignment about machine learning. I'm just trying to rely on their examples with linear regression and information on sklearn

This is from the sgd page in scikit, not sure if I should stick with coef_ or just change it to densify

I just needed some evaluation tools so I went with the coefficients, rsquare and the rmse

[thin palm]

Hi guys, I applied to a Machine Learning Engineer position and the recruiter responded a few days later saying this:
Thanks for your application for the position of Machine Learning Engineer!

I'd like to schedule a call with you. When are you available for a whatsapp call? What about Monday at 3:30pm time?

Looking forward to hearing from you!

What does this mean exactly? Will this be a technical interview from the get-go or a simple phone call? Do I need to be studying like hell in the next 2 days?

[royal crest]

if they say it's a call i'd assume it's a call

but i cannot speak for the cultural embedding

[thin palm]

Correct, but a call about technical skills or more just about their company and why I'd like the role?

[royal crest]

could you not ping me please

i'm not entirely sure how one could conduct a "technical interview" through a WhatsApp call

but again, that might be commonplace where you live

besides, i think this is going off-topic

!rule 7

[arctic wedge]

Rules

7. Keep discussions relevant to the channel topic. Each channel's description tells you the topic.

[thin palm]

I pinged you to let you know that I was responding...

[royal crest]

i'm right here

[edgy brook]

what would you use to evaluate a neural network regression model? I already have RMSE, MAE and R-squared. Are there better evaluation techniques someone can recommend so is the 3 above good? I was thinking SSE but since it's used to calculate RMSE, I don't think its needed???]

[halcyon storm]

Hello

Can someone help me with a project

?

Is it possible if u can help me with my project cuz I need help with a guider cuz I don’t understand. So basically I’m trying to create a algorithm that will detect brain tumors by using a yes n no brain tumor data set. And one of my teachers told me to use a tensor flow guide to follow the steps from there n then do the coding on to the google collaborative notebook

[dull turtle]

i am working with pandas dataframe
i have in that date & time column
in that i have value python bnf_date&time 598 2017-03-02 09:15:59 599 2017-03-02 09:15:59 600 2017-03-02 09:16:58 601 2017-03-02 09:16:59 602 2017-03-02 09:17:41 603 2017-03-02 09:17:59 604 2017-03-02 09:18:54 605 2017-03-02 09:18:59 606 2017-03-02 09:19:57 607 2017-03-02 09:19:58 608 2017-03-02 09:20:58 609 2017-03-02 09:20:59 610 2017-03-02 09:21:05 611 2017-03-02 09:21:59
this way . i want to remove each minute first value for that i get only second value for each minute
so my expected output is python bnf_date&time 599 2017-03-02 09:15:59 601 2017-03-02 09:16:59 603 2017-03-02 09:17:59 605 2017-03-02 09:18:59 607 2017-03-02 09:19:58 609 2017-03-02 09:20:59 611 2017-03-02 09:21:59 this way

ping me when replying

[glacial gale]

Hi can anyone help me with finding a reliable way to analyze football plays based on video footage?

[halcyon storm]

Hey

Guys

[halcyon storm]

Hello

So basically i am working on creating an algorithm that will detect brain tumors with a yes and no response ok using a MVP version for the algorithm. And my teacher told me to firstly import the brain tumor data set onto a new google collab project notebook. And then order me to follow a tutorial guide called tensor flow that will help me out To Split the dataset into training / testing / validation sets which is the other step. However, I don’t comprehend the tutorial guide cuz it seems confusing? Can u be able to help me guide n tell what I should be doing throughout the process

Below is the tutorial guide
https://www.kaggle.com/navoneel/brain-mri-images-for-brain-tumor-detection

Here is the tutorial guide
https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist#1

https://colab.research.google.com

That is the google collaboratory notebook website to get all the work completed

[willow seal]

wong channel

ops

[lapis sequoia]

figure you guys might know, how do i get jupyter's notebook to stop changing the number of loops on each %timeit run?

[cobalt jetty]

%timeit -n <xx> should let you set the number of loops.

though I'd not recommend it in general because timeit fixes that number depending on the length that previous iterations took.

[lapis sequoia]

i see thank you
ya that makes sense but i thought i should know

[tender hearth]

Annoying how torch.nn.utils.rnn.pad_sequence can't pad along an arbitrary axis

Grr

[vast yacht]

Hi i'm new to Data Science and i'm planning on a project that deals with real-time data. How to make the code run non-stop on a specific task? Do I have to put the code in a live hosting site? I have code for crawling products data from a webpage in order to transfer data in a pipe to my database.

[cinder schooner]

hello

any one knows how to use logistic regression to create a scoring method?

[pseudo igloo]

Post it in #help-donut or one of that Chanel

[halcyon storm]

Hey

[random moth]

kind of

#Building a Logistic Regression Model with Sklearn
import pandas as pd
from sklearn.linear_model import LogisticRegression#importing the logistic regression model

df = pd.read_csv("titanic.csv")
df["Male"] = df["Sex"] == "male"

#It is convention to call our 2d array of features X and our 1d array of target values y
X = df[['Pclass', 'Male', 'Age', 'Siblings/Spouses', 'Parents/Children', 'Fare']].values
y = df['Survived'].values

#all sklearn models are built as classes
model = LogisticRegression()#initiating the class
model.fit(X, y)#building the model

print(f"{model.coef_}\n\n{model.intercept_}\n")#the resulting equation is: 0 = 0.0161594x + -0.01549065y + -0.51037152

#we can use the predict method to make predictions
#the syntax is: model.predict(X)
print(model.predict([[2, True, 20.0, 1, 0, 7.25]]))#example, output says whether or not the program expects the person to have died
print("")

n = 14#numper of datapoints to be tested
print(model.predict(X[:n]))
print("")
print(y[:n])


if all(model.predict(X[:n]) == y[:n]):#with larger n, less likely to be completely correct
    print(f"\nThe model was completely correct for {n} datapoints.\n")#14 is the largest n to yield a completely correct result
    
#we can see how good our model is by scoring it based on the number of datapoints correctly predicted, this is called an accuracy score
y_pred = model.predict(X)#an array of the predicted y values
y == y_pred#we create an array of boolean values of whether we predicted each passenger correctly or not
print((y == y_pred).sum(), "\n")#we use the numpy sum method to see how many we got correct

#to get the percent of passengers, we divide the number of correct predictions by the total number of passengers
print((y == y_pred).sum() / y.shape[0], "\n")

#alternatively, we could use the score method
print(model.score(X, y))

this is what I have

[odd meteor]

Can you elucidate?

What do you mean by scoring method? Do you mean metric for evaluating model performance?

[lone drum]

hello i have a dataframe

in one of column i have activity_type values

this way i have my dataframe

i want to keep my activity column in such a way that sell can not came when day is starting

it will only came when previous was buy

means day is start with buy only not sell

if sell is coming at day start them i have to do nothing at that time

if there is buy then only sell will happen

ping me when u reply i will anything other info u need

[next phoenix]

Found this on medium

https://medium.com/coders-mojo/best-collection-of-500-computer-science-courses-with-videos-and-books-ed1d7fe3e751?sk=590c8e677f10e813e24c1c9c8254bf0a

[spiral wolf]

hi, i am about to write my bachelor thesis in hybrid intelligence in the field of Hybrid intelligence/XAI, and my prof told me to play around with python demo data sets if I am going to plan a practical part. I dont know much about python. He suggested me three frameworks, LIME, SHAP and ANCHORS. What exactly can I imagine about play around with demo data sets. and what exactly would be programmed in this domain with python? It's a pity that My programming skills are not the best. Any help is appreciated. 🙂

[quiet vault]

Are u just using pictures to detect the tumor or some other kind of data?

If it is just pictures, this is an image classification problem

Tensorflow is a library used to create machine learning models. There could be a guide on tensorflow but tensorflow is self is not a guide

[hollow shell]

Yolov5 :))

[quiet vault]

Awesome

What library are u using

@hollow shell

[hollow shell]

https://github.com/ultralytics/yolov5
check this

[quiet vault]

Will do

[hollow shell]

Using labelimg you can make your own datasets

[quiet vault]

Yeah I know

What are you using to run

Pycharm?

[hollow shell]

Yep

[quiet vault]

How much vram does it take

[hollow shell]

24

[quiet vault]

Oh

F

[hollow shell]

Coco dataset

:/

[quiet vault]

Have you tested the fps

[hollow shell]

I haven't tried it yet

[quiet vault]

Alright

I really want to play around with it

But I don’t have 24 gigs of vram

Sad

[hollow shell]

Training takes a lot of resources and it takes less than a second to recognize an object.

Use pre-trained models

[quiet vault]

How much vram does it take to make predictions

Same as training?

[austere swift]

no, a lot less

[hollow shell]

It's light enough to be executed in cpu.

[austere swift]

training also needs to store grads and optimizer information, inference just needs the model and sample

[hollow shell]

less than 1 sec for prediction (cpu)

[quiet vault]

0.1

That’s really good

Thanks

[pseudo igloo]

Uhh you can pick up a ripjaw kit around black Friday or used ballistics

[austere swift]

vram not ram

[pseudo igloo]

Oh nvm brain fart

[pseudo igloo]

Hm curious now. Two 3080tis and a nvlink, will that due?

[austere swift]

3080ti doesnt have nvlink

[hollow shell]

3090

[austere swift]

yeah only the 3090 does

but anyways, a single 3080ti will more than do for inference

[pseudo igloo]

Wooow... I'm outdated as hell

[quiet vault]

What do people use to train these models?

[austere swift]

what models?

[quiet vault]

Yolov5

[austere swift]

that can be trained on a single multi-gpu system

[quiet vault]

And others that take a tremendous amount of vram

Ah

[austere swift]

probably one of those 8 a100/v100 servers

those are pretty common

[quiet vault]

Alright

[hollow shell]

It seems to be used in proportion to the vram you have.

(It can be trained on cpu too)

Smaller batch size -> less ram usage

Or small image to learn

[austere swift]

its completely unfeasible to train yolov5 (or any somewhat decently sized model) on cpu

[halcyon storm]

Yeah ur right. Is it possible if u can help me guide through this process plz? I would appreciate so much. I just need to first the
(1) Import the dataset into a fresh Google Collab project
(2) Split the dataset into training / testing / validation sets And that’s all. Can u help me ?

[quiet vault]

Very affordable

[austere swift]

people rarely buy them and run locally, most of the time its provided by cloud

or its large institutions that buy multi million dollar datacenters filled with them

[quiet vault]

Oh

Yeah that makes more sense

[quiet vault]

First go to google drive and create a new document. Instead of selecting one of the most common options (docs, slides) put more and then google colab. Now you can go on and code like normal

I’m not sure how to import the dataset but once you have it imported you have to check to see if the y is in numbers

If it isn’t you have to make it 0 or 1 because machine learning models use numbers

0 for no brain tumors and 1 for brain tumor

After that you can use the train_test_split function from sklearn to split it

[pseudo igloo]

I was helpimg her on that project but the instructions that she was following god knows how out dated it is.

[halcyon storm]

Hey according to my teacher i have to use google collab

[halcyon storm]

I found this video on YouTube can u help me guide https://youtu.be/T8xEQI8XXGs

[quiet vault]

Oh no

[halcyon storm]

Is it possible u can help me guide using that video ? I’m just confused bc i m not an expert at coding

[quiet vault]

Do you know python

You need to know python to do ai

It is way to complicated to learn both at once

[pseudo igloo]

@halcyon storm have you set up the program?

like this?

[arctic wedge]

Hey @pseudo igloo!

It looks like you tried to attach file type(s) that we do not allow (.ipynb). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a.

Feel free to ask in #community-meta if you think this is a mistake.

[pseudo igloo]

huh...

it's a ipynb file....

@quiet vaultdo i have to get permeation for this type of file?

[quiet vault]

Yeah that’s what colab is

[quiet vault]

I’m not sure

[halcyon storm]

Ohh I see . Do u know how long it takes to to complete these first steps

?

[halcyon storm]

No not yet but will do

[quiet vault]

Which first steps

The ones that I told you or learning how to code python

[pastel valley]

guys i am watching about convolutional neural networks
and all about this layers i see that there is something like default layers on tensor flow

but i dont understand like if i like to create a cnn to classify diff types of cars how do i change or what to change on that default cnn?

[pseudo igloo]

uh the plug ins are there on the tutorial page

she don't know how to plug in

[wicked grove]

Hello,I'm stuck on a problem could you please guide me
I have a project where i have to work to with the diabetic retinopathy dataset on kaggle
The training zip has 8k images and a csv file for the labels
It's my first time using jupyter notebook and im really new to deep learning
Could you please tell me if i should follow this code to import the dataset
https://stackoverflow.com/questions/62218611/how-to-train-a-model-with-a-dataset-in-which-image-dataset-is-given-and-label-fo

[quiet vault]

Yeah the code looks good

I would follow it

[wicked grove]

I cant understand why they have used the the tensorflow part tho

I just want to load the dataset and do the preprocessing w opencv

[quiet vault]

You can replace the tensorflow part with opencv

I normally work with tensorflow though

[wicked grove]

Oh okayy, thank you😁
Also, this dataset is for multi class and
I want to do a binary classification
I thought i would get rid of mild column, and combine the other 3 labels(is the approach correct)
But i cannot understand how i can get rid of the mild images

[wicked grove]

Is that better for preprocessing compared to open cv

[quiet vault]

I’m not sure. I have never worked with opencv before. The best tool is the one that works for you

[wicked grove]

Alrightt thankss:))

[cinder schooner]

@random moth @odd meteor

i'm trying to help someone in a research for medecine

and for medecine they use scores like this:

AGE >65 : 2points
AGE<=65: 0 points

DIABETES: YES 1 point
No: 0points

ETC

then they take the sum of the points they get for that patient

and they can tell from the formula if he has big chance or low chance to have the disese or to die etc

and this is given by logistic regression

i found the solution to how to get a score like this from a dataset of patients that have or not the disese

but i dont understand whats the odds ratio for logistic regression

and how to use it for my scoring formula

[odd meteor]

https://www.analyticsvidhya.com/blog/2021/07/what-are-the-odds-and-odd-ratios-that-you-know-about-the-odds/

[cinder schooner]

this is what i always find about odds ratio

but i have a table

with odds ratio of each Beta in the logistic regression

exp(B0),exp(B1) etc

what do i do with that

[pseudo igloo]

@quiet vaultwould you join on a call yo help @halcyon storm ?

[quiet vault]

I can’t sorry

I’m working right now

[odd meteor]

@cinder schooner use the formula in pic to compute your odds ratio. P in the above formula = probability

I presume you're working on a binary classification problem.

1. For class 0 ( No diabetes) , calculate this for your P0 /1 - P0

2. For Class 1 (Diabetes), calculate this as well P1/1-P1

3. Then divide #1 by #2

[pseudo igloo]

alright. I just can't find the file.

[quiet vault]

I’ll help once I’m done

[median fulcrum]

|___| <--- 👆 --->

[halcyon storm]

Hey can u help me thx ?

[halcyon storm]

When ur done can u help. Can u help n join on a hop on a discord call ?

[quiet vault]

I can now

I’m on my phone though

So you should share your screen

Are u in a call with 000

[halcyon storm]

Yes I am

Like now

[quiet vault]

Ok

Add me to the call

[halcyon storm]

Ok bet

[pseudo igloo]

i don't know how to add you.

[sturdy plover]

Where to start with python & ai?
My goal is an ai which detects if a link is malicious
(I already know python)

[halcyon storm]

Hello Ligma

U r added to the group call

[quiet vault]

I would start with a tutorial made by TechWithTim. It explains the concepts and coding on many different models

I’m pretty sure the tutorial is called deep learning for beginners

https://youtu.be/tPYj3fFJGjk

Here is the link

[simple ivy]

hey folks, i just used labelImg to label my data but noticed that it saved the labels as .txt files and im looking at a tutorial that says they should be .xml files, does this mean that ill have to convert the file type or is it okay to proceed with txt?

my assumption is the former, but i wanted to check in here in case anyone has some insight

[wicked grove]

Hello @quiet vault
Could you please help me w this,i am not getting it

[quiet vault]

So you want to turn a multi classification problem in binary classification?

[wicked grove]

Yess

[quiet vault]

So how many classification options are there

[wicked grove]

5 classes

[quiet vault]

And which one are u looking for

Cuz you could do 1 class is equal to 1 and the other 4 are 0

[wicked grove]

This is the dataset https://www.kaggle.com/c/diabetic-retinopathy-detection/data

[quiet vault]

Do you want to do 0 is no dr

And 1 is having dr

[wicked grove]

Yeahh,i want to keep 0 as no and 1 is dr

[quiet vault]

Ok

So

[wicked grove]

Yess

And i i want to get rid ofild

**mild

[quiet vault]

Why

It could be classified as having dr

[wicked grove]

Because the difference is very less
The features of mild aren't as prominent

[quiet vault]

Ok

Makes sense

So do you have this data in a pandas dataframe

https://stackoverflow.com/questions/18172851/deleting-dataframe-row-in-pandas-based-on-column-value

Here’s a link on how to remove rows according to a value in a pandas dataframe

[wicked grove]

I do have the labels in a csv but not the images
Like how do i get rid of the mild images too

[wicked grove]

Thank youu!!

[quiet vault]

Oh wait these are images

Wait nvm

That link won’t work

Honestly I’m not sure how to remove the mild values

[lone drum]

Can anyone help me in this?

[wicked grove]

Oh okay
Also how can i make the other 3 labels as 1

Indicating dr

[quiet vault]

https://datatofish.com/replace-values-pandas-dataframe/

df['column name'] = df['column name'].replace(['2'],'1')

df['column name'] = df['column name'].replace(['3'],'1')

Etc

[wicked grove]

Ahh okayy,thanks a lott!! I will do this

[quiet vault]

No problem

[quiet vault]

I don’t know, sorry

[modern dragon]

Not sure if this is the right channel, but I was wondering if this laptop would be good for machine learning, and Uni:
https://www.bestbuy.ca/en-ca/product/lenovo-yoga-c940-14-touchscreen-2-in-1-laptop-intel-ci7-1065g7-512gb-ssd-32gb-optane-16gb-ram-en/15341476

[royal crest]

probably wiser to use google colab

having a 4K touch screen is not going to help with training models unfortunately

[modern dragon]

What's that?

[royal crest]

Google Colab or Colaboratory is a free cloud platform Machine Learning supported by Google. It allows its users to use free CPU and GPU services. It's great for people and small scale companies working in Machine Learning but doesn't have a GPU lying around. COLAB uses Jupyter Notebook files and does not require any kind of setup.

this sums it up well.

if you're doing a hobbyist level of ML then google colab will suffice

[modern dragon]

What if I want to go into computer science for Uni? or do ML at work?

[royal crest]

my students use google colab for most of their ML work, and if you're doing serious ML as a job, then your work should provide you with the necessary hardware/access

[modern dragon]

Ohh ok thanks!

[last echo]

#help-cookie halp

[proven rock]

hi

[wicked grove]

Before i put the path as an argument for pd.read_csv should i load the dataset in one of the folders of jupyter notebook. Currently the excel file is in my c drive

[lone drum]

Traceback (most recent call last):

File "E:\nifty_banknifty\banknifty_nifty_backtest4.py", line 151, in <module>
condition15 = [end_df3['date'].shift(1) != end_df3['date'] & end_df3['bnf/nf'] > 2.5]

File "C:\Users\Admin\anaconda3\lib\site-packages\pandas\core\ops\common.py", line 65, in new_method
return method(self, other)

File "C:\Users\Admin\anaconda3\lib\site-packages\pandas\core\arraylike.py", line 59, in and
return self.logical_method(other, operator.and)

File "C:\Users\Admin\anaconda3\lib\site-packages\pandas\core\series.py", line 4989, in _logical_method
res_values = ops.logical_op(lvalues, rvalues, op)

File "C:\Users\Admin\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py", line 355, in logical_op
res_values = na_logical_op(lvalues, rvalues, op)

File "C:\Users\Admin\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py", line 272, in na_logical_op
result = libops.vec_binop(x.ravel(), y.ravel(), op)

File "pandas_libs\ops.pyx", line 248, in pandas._libs.ops.vec_binop

File "pandas_libs\ops.pyx", line 241, in pandas._libs.ops.vec_binop

TypeError: unsupported operand type(s) for &: 'datetime.date' and 'float'

How to fix this error

[arctic wedge]

:incoming_envelope: :ok_hand: applied mute to @chilly sable until <t:1635761006:f> (9 minutes and 59 seconds) (reason: duplicates rule: sent 4 duplicated messages in 10s).

[royal crest]

i think the last line of the error message states it clearly

[lapis sequoia]

I want to become data scientist

How shall i start

I am 15 years old and i want to become data scientist in future

[median fulcrum]

|___| <--- 👆 --->

[wide meadow]

Can I choose SVM as base learner in boosting algorithms like adaboost and xgboost?

[tender hearth]

Are padded tokens also positionally encoded in transformers?

[timid rivet]

what're some good (and free) resources to learn data science and machine learning?

i tried datacamp but its not free

but i did like the way their courses were structured

so are there any free websites like datacamp?

[wide meadow]

Statquest's machine learning playlist, Krish Naik's playlist, Hands on machine learning with scikit learn and tensorflow book (structured), Introduction to Statistical Learning book

[wide meadow]

You can try Coursera, they generally provide Financial aid to students

[timid rivet]

but im not a college student, im like, 13 y/o

[timid rivet]

thank you so much :)

i'll check out all of these

[wide meadow]

No problem, you may still get that. Just try applying. You may check Andrew Ng's ML course and Applied Machine Learning with Python Specialization on Coursera. Also, you can freely watch the videos on Coursera (audit the course) but you need to pay for getting the certificate and accessing the assignments.

[timid rivet]

oh ok

thanks again :)

[wide meadow]

No problem :)

[lone drum]

hello my dataframe https://cdn.discordapp.com/attachments/366673247892275221/904397684192976956/unknown.png this way

i want to check if there is new day then buy occur first then only sell occur

no sell occur for every new day

always activity happen buy then sell this way

my code here ```python
end_df3['last_col'] = [(end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)]

but i am getting ```python
Traceback (most recent call last):

File "F:\nifty_banknifty_data\bnf_nf_backtest4.py", line 160, in <module>
end_df3['last_col'] = [(end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)]
File "C:\Users\shubh\anaconda3\lib\site-packages\pandas\core\internals\construction.py", line 751, in sanitize_index
raise ValueError(
ValueError: Length of values (1) does not match length of index (3677)``` this error

my code here https://paste.pythondiscord.com/imisidovod.go

[plush leaf]

Hi, I'll enter a Tensorflow Certification exam at the end of November of the beginning of December. Accordingly, I try to find some examples which I'll used as it's a open-source exam. I've found this kind of resources up till now. Here are the links: https://github.com/nicholasjhana/tensorflow-certification-study-guide , https://github.com/btcnhung1299/tfcert-practice , https://github.com/holic1021/tfcert, https://github.com/GianniVCettolo/DeepLearning.AI-TensorFlow-Developer-Professional-Certificate-program, https://github.com/Tosindare/TensorFlow-Developer-Projects, https://github.com/BankNatchapol/TensorflowCertificated, https://github.com/seanjudelyons/TensorFlow_Certificate, https://github.com/rttrif/TrifonovRS.Deep_Learning_Portfolio.github.io. Would you recommend me other good examples which are straight dedicated to the exam except for them as this exam takes 5 hours and I cannot find enough time to look through all these resources?

[serene scaffold]

why is the whole thing wrapped in [ ... ]

[lone drum]

which line ?

[serene scaffold]

end_df3['last_col'] = [(end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)]

look at [(end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)]. It's a list with one element in it.

and the one element is (end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)

[lone drum]

i am trying to do conditional statemnt

[serene scaffold]

you don't use square brackets to group expressions.

square brackets make a list

in either case, you don't need to wrap the entire right side of the assignment statement

not even with parentheses

[lone drum]

i will explain what i am trying to do i have a dataframe as shown above i want to do that in activity column i want to use condition if it is new date then it will always start with buy and day end will be sell for buy condition bnf/nf > 2.5 this and for sell condition bnf/nf -2.5 this way @serene scaffold this way do u get my point till here

[serene scaffold]

you just need to remove the square brackets from either side of [(end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)], because the square brackets put your expression in a list

and you don't want that.

[lone drum]

end_df3['last_col'] = (end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5)```  u mean this way  ?

[serene scaffold]

yes.

[lone drum]

also i want to write condition for sell in same statement how i can write ?

[serene scaffold]

you can add another expression with &

you seem to have gotten the basic idea with & (end_df3['bnf/nf'] > 2.5)

see if you can come up with the other one

[lone drum]

end_df3['last_col'] = (end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5) & (end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] < -2.5)``` this way

plz check ?

[serene scaffold]

end_df3['last_col'] = (
    ((end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] > 2.5))
    & 
    ((end_df3['date'].shift(1) != end_df3['date']) & (end_df3['bnf/nf'] < -2.5))
)

I would do it like that so it's slightly easier to read

but you could just do

end_df3['last_col'] = (
    (end_df3['date'].shift(1) != end_df3['date']) 
    & (end_df3['bnf/nf'] > 2.5)
    & (end_df3['bnf/nf'] < -2.5)
)

I think.

[lone drum]

also how i can provide statement if end_df3['bnf/nf'] > 2.5 this is true then buy will be written in side column and end_df3['bnf/nf'] < -2.5 then sell will be written in side column

do u get my point here ?

i am getting this way ? @serene scaffold plz check

i am getting False everywhere in last column

[serene scaffold]

sorry, something just came up with work

[lone drum]

okay, do u get what i am trying to do ?

[lone drum]

ping me when u back

[simple ivy]

anyone know the difference between yolo and pascal voc formats for object detection? not finding a lot on the internet

[lone drum]

can anyone help me in python condition14 = [ end_df3['new_col'].astype(bool).shift(1) & end_df3['bnf/nf'].astype(bool) > 2.5 , end_df3['new_col'].astype(bool).shift(1) & end_df3['bnf/nf'].astype(bool) < -2.5] choice14 = ['buy_here_only', 'sell_here_only'] end_df3['new_activity'] = np.select(condition14, choice14, default = 'na') i using this code but not getting expected op

[serene scaffold]

I already explained that you can't wrap expressions with square brackets. That makes it a list.

[lone drum]

condition14 = end_df3['new_col'].astype(bool).shift(1) & end_df3['bnf/nf'].astype(bool) > 2.5 , end_df3['new_col'].astype(bool).shift(1) & end_df3['bnf/nf'].astype(bool) < -2.5
choice14 = ['buy_here_only', 'sell_here_only']
end_df3['new_activity'] = np.select(condition14, choice14, default = 'na')``` this way ?

[serene scaffold]

yes

[lone drum]

i am getting this way this is not expected op

can u plz look into it ?

[serene scaffold]

I don't have time to do that much; sorry

[lone drum]

actually i am at the end of this problem , but got stuck here

if possible can u take look for a min ?

[serene scaffold]

No

[lone drum]

where i am doing wrong

i also can not understand ?

plz for a minute can anyone look here in my problem ?

[robust jungle]

Pycharm’s venv seems to not recognize tensorflow.compat.v1 as a module

[robust jungle]

Sure what is it? No guarantee I know how to fix it though.

[lone drum]

check my code here https://paste.pythondiscord.com/huxonozime.go

[lone drum]

@robust jungle see this

[robust jungle]

So if it’s a new day it returns buy

And if it’s the end of a day it returns sell?

[lone drum]

yes

no new date will happen with sell
and no every date will end with buy
for every buy sell will be there at day end if it does not exists. we can insert dummy row for that case
@robust jungle

[robust jungle]

Alright, what’s wrong with it?

[lone drum]

now i am getting sometimes as at day start sell and day end buy which i do not want

[robust jungle]

Alright

Unfortunately I don’t know how to fix it

[lone drum]

can u please help me later when u get free ?, i got stuck here

[tight walrus]

anyone can help?

[robust jungle]

How can I convert CSV to tfrecord?

[modest timber]

Hi, I have little question about data normalizing and standardization in ML. For example if I'll make LSTM and want to have one feature with data for a day of a week - should I normalize to values between 0 and 1? Or maybe should I make 7 new columns(separate features) with marked 1 for proper day of week ?

[tough bolt]

What clustering algorithms would you recommend to achieve clusters like in this picture?

[cobalt jetty]

If those "strings" are actually series of points, the simplest one you can try is a K-means. A fancier one would be the Expectation-Maximization algorithm with a gaussian mixture model.

[tough bolt]

kmeans what is actually used in the picture. works semi-well.

[tough bolt]

How fancy are we talking? How many hours for the implementation compared to k-means?

[cobalt jetty]

You have pre-made libraries for those -- implementing them from scratch is usually something you do in school as a learning opportunity. And I say fancier because you usually use the former to initialize the latter.

[tough bolt]

oh yeah nevermind. sklearn/scikit seems to have it. should've probably googled before asking. Anyways, thanks

[cobalt jetty]

np

[lyric ermine]

0    "a"        x1 
1    "b"        x2     
2    "c"        x3
3    "d"        x4 
4    "e"        x5  

list = ["a","d"]

->

0    "a"        x1 
3    "d"        x4 

hey all, short question.
so i got a list with some data, i wanna check for each row if values from A are in the list. if not i want them deleted. whats the best way of doing this? iam quite stuck, but i guess you have to do it with isin() somehow.

[cobalt jetty]

if your table is formatted as a numpy array, you can filter your A column using the list as a check. You get out a list of true or false that corresponds to which rows you want to keep.

import numpy as np
array = np.array([["a", 1], ["b", 2], ["c", 3], ["d", 4], ["e", 5]])
check = ["a", "d"]
array[[x in check for x in array[:,0]]]

if your table is a pandas dataframe, look up how to use .apply() for instance.

a different way to express this would be by using the standard library function filter. Reusing my previous two variables:

np.array(list(filter(lambda x: x[0] in check, array)))

hoping it helps.

[lyric ermine]

ty @cobalt jetty

[cobalt jetty]

np

[modest timber]

Hi, should all data must be normalized?

in ML

[simple ivy]

hey all, can you do data augmentation on labeled images?

my assumption is yes, but i suppose my question is how it impacts the .xml/txt files for the labels if there's data augmentation

[quiet vault]

It should for most cases

[quiet vault]

Why is this happening

Look at the val_binary_accuracy

[serene scaffold]

It's not likely that anyone will read this photo. Try opening Discord on the computer where this is happening and copying/pasting the text.

[quiet vault]

Bruh

[quiet vault]

So it’s the best it can possibly be

[royal crest]

with what you have yes

[quiet vault]

I see

That good

And bad

Thanks

[austere swift]

it's overfitting, not plateauing

@quiet vault overfitting bad

plateauing is when it converges then just stays at the same loss/accuracy (in other words, it's done training). overfitting is when it starts to "memorize" the training samples, so it does really well on the training set but very poor on the validation set

you can see the validation loss starts going up after epoch 8

so the model is overfitting

[wicked grove]

Can someone please give me a review for this
I did a small twitter sentiment analysis project following a tutorial ,here is the code and my output. I tried to match the code to the tutorial yet my accuracy differs from theirs

https://paste.pythondiscord.com/ocosulaxef.py

           precision    recall  f1-score   support

           0       0.77      0.77      0.77     39752
           1       0.77      0.77      0.77     40248


    accuracy                           0.77     80000
   macro avg       0.77      0.77      0.77     80000
weighted avg       0.77      0.77      0.77     80000

[wicked grove]

My accuracy isn't that good and i wanted to know if there are ways to improve it.

[quiet vault]

Ah

Makes sense

[gleaming axle]

Hey there! I'm extremely new to the whole AI/ML scene, so please go a bit easy on me 😅

    I was experimenting with GPT-2 to use for my final-year project, and wanted to build a working prototype in a short time. I tried the vanilla model, but it obviously spewed random (although coherent) bullshit a few sentences in.
    So I tried to fine-tune it with this library called aitextgen and I followed the instructions from this Colab notebook: https://colab.research.google.com/drive/15qBZx5y9rdaQSyWpsreMDnTiZ5IlN0zD?usp=sharing, and trained it with this (https://www.kaggle.com/jrobischon/wikipedia-movie-plots) dataset from Kaggle (wikepedia movie plots).
    The idea is to provide a movie title as prompt, and have it generate a plot summary, along with other details like year, etc. But somehow, the model seems to be only considering the first column Year, since my generated output consists solely of a bunch of years appended together.

How do I format the CSV file, or how do I tell the model that it should consider all the columns for the training?

[pastel valley]

the the inputs for convnets should be same sizes?

[cobalt jetty]

They usually should because your convolutional neural networks need inputs of the same sizes. If your inputs are of varying shapes, consider using data augmentation (e.g. sheering, cropping, etc.).

[shut trail]

you didnt ask a question

first one is a dict with string and int entries, the other uses lists ... i duno

[tacit spade]

I claimed a help channel

[shut trail]

uh huh cool 🙂 good luck

[native solstice]

Hey, anyone able to run this project? Its not working for me. I cant run all the code without error

https://github.com/erogol/beauty.torch

[shut trail]

I havent used this library but giving specifics about your problem is more likely to attract someones curiosity

you have Torch working ?

[native solstice]

@shut trail loafing up my Linux machine now to answer both questions - one moment please

1. I'm using Jupiter notebook and python 3 in it.

2. I believe I installed torch by using the instructions in the link that the project provided http://torch.ch/docs/getting-started.html

3. This is my current error from exampleuse.ipynb

Note: I had to install Lua at some point and ended up changing the source code when things weren't working so if you recommend I start from scratch and say what the errors are then that's fine

@shut trail Note: that error is comming from the first block in the exampleuse file

@shut trail I'm also willing to run the project in Google collaboration or anything where you think I can get it working....I basically want to just run the project successfully (the mechanism doesn't matter to me).

[native solstice]

Maybe I should use a different example. Basically I want to use a project that uses machine learning to rate a selfie image from a scale of 1 to 10 like a human would.

[shut trail]

you got a menu over the pic you sent

[pastel valley]

i see thank you sir

[fathom tiger]

Hello, So I’m trying to multiply a matrix A of shape (Batch_size, n_steps, n_features) with a vector V of size n_steps where each each element at position i of V is the weight of A at ( ith_n_step x n_features )

Anyone have an idea on how to multiply these ?

I have tried np.matmul() but gives me a ValueError

[fathom tiger]

A looks like this: <tf.Tensor: shape=(2, 6, 5), dtype=float32, numpy=
array([[[-3.8469510e-04, 4.8987731e-05, 1.0081288e-02, 6.9628754e-03,
-2.5918424e-02],
[-1.9368050e-03, 1.0827677e-04, 1.3968085e-03, -1.3400731e-03,
-1.9538682e-04],
[-1.7001404e-04, 1.3932746e-05, -6.3095507e-03, -5.8138347e-03,
1.8222772e-02],
[-6.0863770e-03, -1.6360680e-03, -7.4422453e-03, -1.4341861e-03,
9.0264846e-03],
[ 1.1787540e-02, 1.8913264e-03, 1.5856372e-03, -5.9261476e-04,
3.2372773e-03],
[ 8.3818398e-03, 1.8284719e-03, 9.1116680e-03, 3.2469342e-03,
-1.2978968e-02]],

   [[ 9.5181176e-03,  1.9850563e-03,  7.5515169e-03,  1.4524567e-03,
     -8.1461631e-03],
    [ 3.4734907e-03,  1.2104939e-03,  1.0334797e-02,  3.1762398e-03,
     -1.6403435e-02],
    [-1.1904579e-02, -1.7603291e-03,  5.2853790e-03,  4.1572955e-03,
     -1.8757440e-02],
    [ 5.9668538e-03,  1.5568929e-03,  1.0137518e-02,  1.7998712e-03,
     -1.4689988e-02],
    [-1.3885480e-02, -2.2563422e-03,  3.0955765e-03,  3.2448841e-03,
     -1.6058493e-02],
    [-1.6448567e-02, -3.2972097e-03, -9.1640605e-03, -2.1260502e-03,
      6.6152485e-03]]], dtype=float32)>

[serene scaffold]

I was trying to figure it out and then my boss randomly called me. The information you gave initially is sufficient

[serene scaffold]

try A * V[None, :, None]. But try to verify that this not only "works" but does what you expected.

[thin palm]

can anyone tell me what the heck
drop='if_binary' does in a One Hot Encoder?

ohe = OneHotEncoder(sparse = False, drop='if_binary)

[fathom tiger]

Thanks alot Sir!!! The output is a tensor of shape (1, 1,1, 6, 5) on an example matrix A.shape -> (1, 6, 5) and V.shape -> (1, 6, 1). So I'd just reshape the output! Thanks @serene scaffold : )

[serene scaffold]

that doesn't seem right?

In [38]: arr.shape
Out[38]: (4, 5, 6)

In [39]: vec.shape
Out[39]: (5,)

In [40]: (arr * vec[None, :, None]).shape
Out[40]: (4, 5, 6)

[fathom tiger]

Yeahh.. my vec.shape was (1, 6, 1)

[serene scaffold]

unless otherwise specified, a vector is one-dimensional.

[fathom tiger]

My bad… Noted!

[serene scaffold]

though an array of shape (1, n) is a row vector and (n, 1) is a column vector. I'm not sure if there's a term for (1, n, 1). It might be some special type of vector (as far as terminology goes)

[shut trail]

its a vector in R3

lets not start maths 😄

[fathom tiger]

Actually curious to know! Was thinking, since 3D is just a stack of 2D, (1, n, 1) could be seen as (n, 1) in 2D, just adding an extra dim(1) makes it 3D. Could be it considered a vector(special type) ?

[fathom tiger]

😅

[shut trail]

youre right haha, its the context

[fathom tiger]

Makes sense 😅

[velvet thorn]

there is no equivalent term for "all-but-one-dimension-are-singletons tensor", AFAIK

[stoic musk]

doing first attempt at YOLO architecture

I've got a 19x19x5x80 tensor, where
indices 0 & 1 are the dimensions of the output grid cell,
index 2 = number of anchor boxes, and
index 3 = number of classes, with each value representing the probability the cell contains an object

I've got the indices of the classes with the largest scores:
box_classes = tf.math.argmax(box_scores[3])

Now I'm trying to get the corresponding values to those indices. any ideas?

[stoic musk]

New question:

If I have a 19 x 19 grid, with 5 anchor boxes for each grid cell, with anchor box containing the probability of an object fitting each box,

How do I create a 19 x 19 x 5 tensor, with each value in the tensor being a 'bool', if the value in the previous tensor exceeds a theshold value (i.e. 0.5)?

[plucky willow]

I am wanting to use sklearn to predict a trend in data, but I do not know how

I am trying to predict the LanduagewantToWorkWith from these columns

Index(['Country', 'EdLevel', 'YearsCode', 'LanguageHaveWorkedWith',
       'LanguageWantToWorkWith', 'DatabaseHaveWorkedWith',
       'DatabaseWantToWorkWith', 'PlatformHaveWorkedWith',
       'PlatformWantToWorkWith', 'WebframeHaveWorkedWith',
       'WebframeWantToWorkWith', 'OpSys', 'Age'],

[tidal bough]

new_tensor = prev_tensor > 0.5?

[plucky willow]

and so far

this is what the code looks like

import pandas as pd
from sklearn import tree
from sklearn.model_selection import train_test_split

"""
TODO: Use PostgreSQL

NOTE: I am reading from the disk instead of the database because school wif is
not working.
"""

def run():
    # use pandas to read a csv file
    print('using pandas to read csv')
    data = pd.read_csv('./data/so-survey-2021.csv')

    # remove unnecessary columns
    print('removing unnecessary columns')
    data = data.drop(columns=['ResponseId', 'MainBranch', 'Employment',
        'US_State', 'UK_Country', 'Age1stCode', 'LearnCode', 'YearsCodePro',
        'US_State', 'Age1stCode', 'LearnCode', 'YearsCodePro',
        'DevType', 'OrgSize', 'Currency', 'CompTotal', 'CompFreq',
        'MiscTechHaveWorkedWith', 'MiscTechWantToWorkWith', 'ToolsTechHaveWorkedWith',
        'ToolsTechWantToWorkWith', 'NEWCollabToolsHaveWorkedWith', 'NEWCollabToolsWantToWorkWith',
        'NEWStuck', 'NEWSOSites', 'SOVisitFreq', 'SOAccount', 'SOPartFreq', 'SOComm',
        'NEWOtherComms', 'Gender', 'Trans', 'Sexuality', 'Ethnicity', 'Accessibility',
        'MentalHealth', 'SurveyLength', 'SurveyEase', 'ConvertedCompYearly'])

    # print column names
    print('working with these columns')
    print(data.columns)

    # print head of data
    print('printing head of data')
    print(data.head())

    # get the most wanted from the data
    most_wanted_languages = data.drop(columns=['LanguageWantToWorkWith'], inplace=True)



if __name__ == '__main__':
    run()

what is the next step that I should take?

I am thinking KMeans and then graph the clusters, but that did not work when I tried earlier

it kept complaining how my data were strings and not ints

when I run the code as is I get this output

using pandas to read csv
removing unnecessary columns
working with these columns
Index(['Country', 'EdLevel', 'YearsCode', 'LanguageHaveWorkedWith',
       'LanguageWantToWorkWith', 'DatabaseHaveWorkedWith',
       'DatabaseWantToWorkWith', 'PlatformHaveWorkedWith',
       'PlatformWantToWorkWith', 'WebframeHaveWorkedWith',
       'WebframeWantToWorkWith', 'OpSys', 'Age'],
      dtype='object')
printing head of data
                                             Country  ...              Age
0                                           Slovakia  ...  25-34 years old
1                                        Netherlands  ...  18-24 years old
2                                 Russian Federation  ...  18-24 years old
3                                            Austria  ...  35-44 years old
4  United Kingdom of Great Britain and Northern I...  ...  25-34 years old

[5 rows x 13 columns]

I guess I should also put the first row as the column names instead as a row, but idk how to do that

[odd meteor]

I thought you mentioned you wanna predict trend in your data? Isn't that suppose to be a Time Series analysis (predicting seasonality & trend) ? Why KMeans?

[plucky willow]

well I was thinking the trend, but then I realized that this data was taken from a 2021 survey and i dont want to make the other years clean

so I was thinking seeing KMeans then ig where I could fill out a person and get their fav lang

maybe later I will make the other like 5 years clean or smth and then see the trend and predict the next years most fav langs

[old grove]

In ttest how to implement one tailed test.. Like say I want to test whether or not the starting age range is 27 or say like age >=27, So in python ttest_1samp,How to put the value of popmean ??? Pls help

[tardy scarab]

    from tensorflow.python.keras.application import MobileNetV2
ModuleNotFoundError: No module named 'tensorflow.python.keras.application'

tf version is 2.6.1

[serene scaffold]

try tensorflow.keras.applications

as shown here: https://www.tensorflow.org/api_docs/python/tf/keras/applications

[serene scaffold]

I have

import warnings 
warnings.filterwarnings('ignore')

in the first cell of my notebook, but warnings still appear.

[quasi parcel]

is there any way to get the data from s3 faster using pyspark

i tried s3n://

spark.read.json('s3n://')

but its too slow

is there anyway

[warm wyvern]

Hello! I have a project and I use pandas dataframe and pandas_ta. Is possible to use numba or something else to speed up the calculations? I needed to test over 5000 000 variations

[serene scaffold]

You will need to be more specific about what calculations you are trying to do.

[warm wyvern]

I'm Sorry. There is a dataFrame - 7 columns, 1500 rows. I calculate ta.tsi - true strength index, and ta.stoch - stochastic

[serene scaffold]

I don't know what those are, unfortunately

[warm wyvern]

I wanted to use numpy but I cant use calculations offered by pandas_ta

[serene scaffold]

pandas is numpy under the hood--anything you can do with a numpy array, you can do with a dataframe (after converting it to an array and possibly doing some reshaping).

[warm wyvern]

dataframe --> numpy array is not a problem, but the result of using numpy array as source of data, instead of dataframe, is "None"

[serene scaffold]

can you show the code?

!code

[arctic wedge]

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

[warm wyvern]

I will make an example, just a moment. because the final program is too complex

import pandas as pd
import pandas_ta as ta

dataFrameCombination = pd.read_csv('combination.csv') # load data to dataFrame

ema1 = ta.ema(dataFrameCombination['2'], 10) # calculate exponential moving average
ema1LastNumber = ema1.values[-1] # = 9.998

numpyArray = dataFrameCombination.to_numpy()
ema2 = ta.ema(numpyArray[:,0], 10) # = None 

[edgy stream]

Hi everyone. I am now thinking of developing an ultra-realistic AI that allows us to simulate dubbing voices that we hear in movies, TV series, video games... Although Python is great for AI, SSML has been suggested to me, which also includes a 'neural network' system. Which of the two languages should I choose?

[serene scaffold]

By SSML, are you referring to the speech synthesis markup language?

I would also temper your expectations with developing an "ultra-realistic voice dubbing AI", as realistic-sounding artificial voices is a technology under active development, and requires collaboration between both AI and speech specialists.

[sour mango]

I want to stop jupyter from opening a new tab every time I open a new notebook, I edited the custom js file but now it sometimes opens a new tab and sometimes doesn’t

Is there a better way?

[median fulcrum]

Please, if someone know....

I need to do this until friday

[edgy stream]

Correctly @serene scaffold, I'm referring well to: Speech Synthesis Markup Language.

[serene scaffold]

SSML is not a programming language. It's a markup language for specifying how you want a given text to be spoken by a synthetic voice. You cannot make programs with it, much like how you can't make programs with HTML (hyper text markup language).

[edgy stream]

So, iirc I can't implement SSML in a GUI app coded in Python, C++...? I understood this well?

[serene scaffold]

what do you mean by "implement SSML"?

SSML is for annotating text.

[edgy stream]

Sorry, I didn't explained me well before 😅

Example: If my app is written in C++, the voices are written in SSML. The written voices in SSML can work in a GUI app coded in C++?

Idk If I've now explained me 🤔

[serene scaffold]

Synthetic voices aren't written in SSML, either.

[edgy stream]

Ah ok, hypothetically, in the case of my desktop application, it would not make sense to write the voices in SSML, but in a suitable and good language for AI like Python. Right? 🤔

[serene scaffold]

You can't make voices in SSML. SSML is just for specifying how you want an existing voice to say something.

Whoever just commented and deleted it, you don't want to call fit_transform twice.

@shy nimbus

[shy nimbus]

i resend my question in help-orange

[median fulcrum]

please........

[lapis sequoia]

how can I pass centers manually in KMeans? I don't understand how to add them
https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

[cobalt jetty]

you pass an array to it with the necessary shape.

the array is in all likelihood a numpy array.

[real wigeon]

i have a bar chart in matplotlib, it splits up some data, i need to somehow also include the total somewhere?

like my N

[strange leaf]

Does anyone know how to use robot studio?

[serene scaffold]

it's best to just ask your actual question, though I would make sure that it realtes to the channel topic.

[strange leaf]

i have problem with sensors

[median fulcrum]

pleaseeeeeee

[thorn canopy]

dont be that guy, anyways search for object recognition.. one of the bigger companies (maybe microsoft, not sure) launched an api that calls a model to detect objects (from phones, to cats, to people etc) .. try searching for that

[median fulcrum]

Thanks for the answer! Sorry for being little bit rude, but I have to do the job until friday and I'm searching for help since 10/29

[thorn canopy]

it is definitely a hard problem that people might not know about, so try asking in different forums / stack overflow, etc

not every question in this server will get an answer, and one should not expect to get one either

[serene scaffold]

I suspect that your question is going unanswered because people have to go back a few jumps to get to it (they're not likely to even follow the first jump) and the question doesn't have anything that's actionable for anyone who doesn't consider themselves a domain expert.

what do you mean by "facial recognition of people even without classifying who the person is". Are you just detecting when there is a face in a picture?

[median fulcrum]

detecting for example harry potter characters, and the algorithm tell Ron or Harry

[serene scaffold]

so this is facial recognition, yes?

[median fulcrum]

I think so, but it's just images with a train database and test

[serene scaffold]

have you looked into torchvision?

[median fulcrum]

No....

[rough mountain]

I keep getting this error when I import keras.
AttributeError: module 'tensorflow.compat.v2.__internal__' has no attribute 'register_clear_session_function'

how can I fix it?

[serene scaffold]

Please show the whole error message.

!traceback

[arctic wedge]

Please provide the full traceback for your exception in order to help us identify your issue.

A full traceback could look like:

Traceback (most recent call last):
    File "tiny", line 3, in
        do_something()
    File "tiny", line 2, in do_something
        a = 6 / b
ZeroDivisionError: division by zero

The best way to read your traceback is bottom to top.

• Identify the exception raised (in this case ZeroDivisionError)
• Make note of the line number (in this case 2), and navigate there in your program.
• Try to understand why the error occurred (in this case because b is 0).

To read more about exceptions and errors, please refer to the PyDis Wiki or the official Python tutorial.

[rough mountain]

  File "C:\Users\\Desktop\Projects\APICleaner\model.py", line 1, in <module>
    from keras.models import Sequential
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\__init__.py", line 25, in <module>
    from keras import models
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\models.py", line 19, in <module>
    from keras import backend
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\backend.py", line 298, in <module>
    tf.__internal__.register_clear_session_function(clear_session)
AttributeError: module 'tensorflow.compat.v2.__internal__' has no attribute 'register_clear_session_function'```

not sure this helps

[serene scaffold]

try from keras import Sequential

actually, from tensorflow.keras import Sequential

[rough mountain]

That one works, but is deprecated

[stoic musk]

AttributeError: 'MaxPooling2D' object has no attribute 'op'

Anybody familiar with convolutional neural networks?

[neon heart]

what does this point to
df.iloc[:,1:])

[stoic musk]

should be all entries except for the first column

[neon heart]

first column as in index or no?

[stoic musk]

depends how the data is structured, but my guess is no

[neon heart]

gotcha, thanks

[stoic musk]

you can find out quickly with df.head

or df.describe

[neon heart]

`
Gender Age EstimatedSalary Purchased
0 0 19 19000 0
1 0 35 20000 0
2 1 26 43000 0
3 1 27 57000 0
4 0 19 76000 0

[stoic musk]

first column 'gender' is not the index

[neon heart]

So - df.iloc[:,1:]) would not include Gender column

[stoic musk]

give it a try 🙂

test = df.iloc[:,1:]

test.describe or test.head

[neon heart]

Ah thanks, I'm rough with indexing, usually type in headers manually, need to find chart/cheatsheet

[stoic musk]

Don't worry if it's important you'll remember

[rough mountain]

My neural network somehow got an accuracy of 20% in a binary classification task. So I guess I take the opposite of what it says and I got 80?

[royal crest]

[quiet vault]

i am

[quiet vault]

It literally takes skill to be that bad

[crude needle]

Hi guys, I want to read multiple csv files and concatenate them like this in a different row. How can I use glob command so that it can automatically sort the data as shown below.

I am using pandas

Actually, the original file is this which has a "p" heading at the last and I have a hundred of such files. I just want to extract this data and do the operation.

[night dragon]

it only needs the column with header 'p'?

[crude needle]

yes sir