data-science-and-ml

[misty flint]

one of my classmates loved it

[paper lake]

i am in a julia server

[misty flint]

[paper lake]

i can invite u :((

[misty flint]

maybe later

im barely learning python

[paper lake]

nooooooooooo

[paper lake]

julia's syntax is as easy as peethon

[misty flint]

nice spelling

[paper lake]

sadsuu i guess i will just leave a link here then

https://discord.gg/SMYjYwK57h

[misty flint]

@hollow sentinel whats the thing youre learning after ds&a

i feel like youre never getting to ML at this rate

or do you want to go the SWE route

[paper lake]

whats SWE?

[odd lion]

Software Engineer

[misty flint]

^

[paper lake]

whats W?

[misty flint]

uhh

i guess the Ware part?

[odd lion]

Soft"W"are Engineer

[misty flint]

[bronze skiff]

winning

[misty flint]

💀

[hollow sentinel]

the math

[paper lake]

frick u love that sticka so much

[misty flint]

"softwinning" ah yes

[hollow sentinel]

it's just linked lists slowing me down

[misty flint]

"the math" dudeeeee

thats almost as big as ds&a

[hollow sentinel]

just practical statistics behind ML

[misty flint]

[hollow sentinel]

just that for now

[misty flint]

oh ok

also its not just only stats but multivar calc but yes

[hollow sentinel]

statistics is one of the biggest parts of ML

so it would be nice if I can knock that out

[misty flint]

yes yes

[hollow sentinel]

plus I have you to help

[paper lake]

thats good. even a bit of knowledge with it can help u like bayesian and stuff but yeah

[hollow sentinel]

and the rest of this server

[misty flint]

plus I have you to help
who

jk

[misty flint]

do you do a lot of bayesian stats with bioinformatics too?

[paper lake]

are you all undergrads?

[misty flint]

das is

im old

🧓

[hollow sentinel]

I'm pretty old dude

I'm turning 20 on april 5

[misty flint]

not as old as me i promise

[hollow sentinel]

28?

[paper lake]

i am just a microbiologist undergrad so probably yes to that i guess or can u share your knowledge sensei

[misty flint]

[paper lake]

[misty flint]

close

[odd lion]

If you're 20, you're not old

[misty flint]

nice! i did molecular bio as my undergrad

[odd lion]

But I remember how it felt being 20 relative to 18 year olds

[hollow sentinel]

I live in a pretty young neighborhood and my parents are like go play w the high schoolers

I mean

a 20 year old playing with high schoolers????

uhhhh

sus

[paper lake]

freak so i am sure i can ask some guidance here from u while suffering with the rain of kekw and pepe stickas

[misty flint]

haha yes

ig if youre going the bioinformatics route, try to do well in your stats class

i regret not taking mine seriously

[hollow sentinel]

speak for yourself I got a A+ in statistics bc I cheated

[paper lake]

i only passed :3 freaking hell i also regretted

[hollow sentinel]

it wasn't just me cheating

it was the entire class

we were all in one groupchat just cheating

[misty flint]

and now you have to learn stats on your own

[hollow sentinel]

well I know some stuff

but it was business statistics

not "machine learning statistics"

[paper lake]

dont worry, i have a book :3 and then i will suffer from it

[misty flint]

same tbh

[hollow sentinel]

rex can I send the book to you

[misty flint]

already suffering

[hollow sentinel]

see what you think

[misty flint]

what book

i also have a couple books that i havent looked at yet

[hollow sentinel]

the practical statistics behind machine learning book

[misty flint]

sure

[paper lake]

imma check my lib again if i had such book

[hollow sentinel]

https://math2510.coltongrainger.com/books/2017-bruce-and-bruce-pratical-statistics-for-data-scientists.pdf

here you go guys

[misty flint]

oh its an oreilly book

nice

much more digestible

[hollow sentinel]

yes

o'reilly books are bae

they're concise

[delicate lintel]

any of you guys learned ai through the internet?

[hollow sentinel]

there are some good resources out there

[delicate lintel]

i wanna learn ai with python but i'm too young to go to uni where can i learn that stuff?

[hollow sentinel]

Kaggle

[delicate lintel]

kaggle?

[hollow sentinel]

yes

it's a site for data science/machine learning

[delicate lintel]

is that free?

[misty flint]

content looks relatively solid

[odd lion]

I'm not sure Kaggle is the best "learn AI from nothing" site. It's good if you have some exposure. The traditional rec is Andrew Ng's Machine Learning course, but I suspect you don't have a great math background yet. What's your math like and how strong are you in python?

[misty flint]

the code is in R but what stats book isnt

[paper lake]

oreilly is mostly good

[delicate lintel]

very strong in python but don't have a crazy math background

[hollow sentinel]

please don't just do Ng's class without knowing the math

it will be hell

[odd lion]

Kaggle is a data science competition website where you are given datasets and need to extract something from them (e.g. a class, predicted price, etc...) There are a number of notebooks where people step through what they did, but I don't think those are a best first resource

[hollow sentinel]

it assumes you know calculus, linear algebra, statistics, probability, and discrete maths

it's also in Octave, not Python

[odd lion]

Agreed, hence why I was hesitating to recommend it

[paper lake]

nice i can just translate some stuff into julia

[delicate lintel]

define descrete math?

[paper lake]

yeah nice too

gnu octave

[delicate lintel]

i know all of that other stuff but i don't know what descrete math means

[misty flint]

youtube. then the algorithm will recommend ML stuff automatically for you

[hollow sentinel]

Discrete math is the math of finite things (as opposed to infinite or continuous elements).

that's the most ELI5 explanation I could find

[delicate lintel]

oh yeah i did that in school

[paper lake]

i think i wont worry much. highschool destroyed me anyway with these stuffs

[odd lion]

To be fair, Discrete math is infinite, just not continuous

[hollow sentinel]

it's a generalized explanation

[odd lion]

Sure

[misty flint]

discrete math is the miscellaneous math subject where they throw all the subtopics from other maths they think engineers might need

[delicate lintel]

like "find the limit of 5/x" and stuff like that?

[paper lake]

frick

[misty flint]

[hollow sentinel]

find the limit of 5/x would be calculus, no?

[misty flint]

amirite or amirite?

[paper lake]

claculusp

[misty flint]

[paper lake]

lol my phone skid

[hollow sentinel]

if you want to learn calculus

[delicate lintel]

i don't know how things are called in my language

[hollow sentinel]

watch 3blue1brown

[odd lion]

Anyway, back to the question at hand, these resources are pretty good for learning AI: https://github.com/louisfb01/start-machine-learning-in-2020

Oh right, I forget Kaggle had courses now, so I take back what I said, you could learn there

[misty flint]

hmm i should do those 100 numpy exercises some time

[delicate lintel]

don't youtube have good ai courses?

[odd lion]

If you really want to learn AI/ML, you're going to need to pair learning AI/ML things like scikit/random forest and studying a lot of math

[misty flint]

itll go on the neverending list

[delicate lintel]

the one i found was techwithtim and i don't like him

[odd lion]

Probably. Many of the courses host their videos on YouTube. I personally prefer more structured coursework than straight YouTube lectures offer but to each their own

[paper lake]

https://github.com/mitmath/18337

[misty flint]

oh definitely find someone you like or else itll be harder to learn

[paper lake]

goood stufff

[hollow sentinel]

I tried doing the MIT algos/DS course

[paper lake]

i like chris

[hollow sentinel]

and it started w peak finding....

not even big O

just peak finding...

MIT is gonna MIT huh

[paper lake]

not alll

just teachers

[delicate lintel]

do i need to learn ds?

[paper lake]

¯_(ツ)_/¯

[delicate lintel]

ds is so boring

[hollow sentinel]

I tried reading the notes and watching the lectures

I got nowhere

[delicate lintel]

do i need ds for ai?

[paper lake]

yeah thats not MIT just the teacher

[grave frost]

you mean data science or data structures?

[hollow sentinel]

well yes

[odd lion]

IF you want to be effective, yes

[paper lake]

yup

[delicate lintel]

data structures

[hollow sentinel]

yes

[delicate lintel]

i do that stuff at school

[odd lion]

Oh, you mean Data Structures? Yes, very much so

[delicate lintel]

it's so boring

[grave frost]

ye, gotta knew atleast the basics

[hollow sentinel]

otherwise the only structures you know are lists and dictionaries

strings?

[misty flint]

yes, it is. thats why im procrastinating on learning it

[hollow sentinel]

which is why I'm learning it

[paper lake]

time to stop procrastinating

[misty flint]

[paper lake]

[delicate lintel]

i know queues trees, linked lists all that stuff

[grave frost]

<strikethrough>I don't know HTML</strikethrough>

[odd lion]

Hash tables?

[delicate lintel]

no

[hollow sentinel]

aren't dictionaries just optimized hash tables

[delicate lintel]

next year we do hash tables and big O

[grave frost]

ur in college?

[odd lion]

Yes, but there's a difference between calling dict and knowing how it works

[delicate lintel]

highschool

[hollow sentinel]

interesting

[grave frost]

wait, they teach you that in HS?

[misty flint]

this is in julia! i see what youre trying to do

[delicate lintel]

yeah

[grave frost]

thats dope man

which country?

[paper lake]

it is called baiting

[delicate lintel]

israel

it's not dope it's boring

[grave frost]

atleast they are teaching you the stuff 🤷

[hollow sentinel]

I think AP CS A in the US teaches algorithms

[misty flint]

its ok. just breeze through the boring stuff for now and come back to it when you REALLY need it

[hollow sentinel]

but they forget them bc they don't bother to look it over till college

[delicate lintel]

ok i will

[grave frost]

indian python CS is pretty basic

[delicate lintel]

we have pretty advanced stuff in cs class compared to other countries i think

we also have to do a graded project next year

[grave frost]

ye, that's in EU too

[delicate lintel]

some kid did an operating system

[grave frost]

seriously??

[delicate lintel]

yeah

it does basic stuff

[paper lake]

ooh nice

[grave frost]

like siraj Raval ? 🤣

[delicate lintel]

who is siraj raval?

[grave frost]

nevermind

[hollow sentinel]

oh god

Siraj Rival is this guy who claimed he was a ML prodigy but all he did was copy code

[paper lake]

🤔dk who that is

[grave frost]

yeah, he also claimed he built an OS

[paper lake]

aaah a script kiddie

[hollow sentinel]

basically yes

[paper lake]

like me

[grave frost]

which was also copy/paste

[paper lake]

[hollow sentinel]

everything he did in his videos

he just copy pasted from someone else's github repo

it wouldn't be a problem if he cited them

[grave frost]

being a script kiddie in general is pretty bad, he made it worse by PLAGIRARIZING

[paper lake]

uhm wait, he read code, copied it, never learned it? what a scam lol

[hollow sentinel]

but he didn't cite them

[grave frost]

yeah

[hollow sentinel]

like devs integrate stack overflow code into their stuff

but they know how it works

they know what it does

[grave frost]

so basically copy the whole repo without creditin

[hollow sentinel]

basically yes

dude dug himself a hole so deep he couldn't think

[paper lake]

what happened after he was caught doing that

[hollow sentinel]

uh all of his credibility vanished

he even ran a course and it scammed hella people

incorrect explanations

[grave frost]

and earned 200k off it

the course was titled "how to get rich with ML"

[paper lake]

freaking hell... Fake Gurus

[grave frost]

so he demonstrated it himself

[hollow sentinel]

he released his own "research papers"

[misty flint]

[grave frost]

oh yeah, that one too

[hollow sentinel]

but all he did was reword what they said dumber

[misty flint]

y i k e s

[paper lake]

💩

[hollow sentinel]

and what he couldn't understand he completely removed

[grave frost]

something like hilbert complex to 'hilbert complicated'

[misty flint]

why do people do this? do they really think they wont get caught?

[grave frost]

which aparently doesn't make sense as its some mathematical term

[hollow sentinel]

at least I know that I don't know DS/ML

and I don't pretend to

[paper lake]

sometimes if lucky, they wont

[grave frost]

true

There are so many get rich quick gurus out there

[hollow sentinel]

well now we have cancel culture

so all the gurus get exposed and canceled

[misty flint]

[paper lake]

were u the one who sent a yt link about the rise of fake gurus

[hollow sentinel]

yep

another problem is these damn data science bootcamps

[paper lake]

i see my useless long term memory can still recall

[grave frost]

and there are more; techlead, joma tech etc.

[hollow sentinel]

Fluke too

Joshua Fluke

he has these "courses" you can buy to start your own company

what bullshit

[paper lake]

i convinced people not to trust joma tech

[grave frost]

was he involved in something

[hollow sentinel]

Joma tech is a sellout

[grave frost]

?

[hollow sentinel]

so is tech lead

[paper lake]

still they say he is cool and trusted

[hollow sentinel]

they both doxxed a kid

[misty flint]

[grave frost]

tho joma has good potential honestly

his vids are a++

especially the startup ones

[paper lake]

cringey enough they made people trust them frick

[hollow sentinel]

tech w tim any day over these people

[primal tulip]

I'm a self taught programmer working as a Data Analyst. I graduated in Economic, but tossed it all when I got bored.

[misty flint]

i watched a guy tear down joma for how he did his amazon interview

[primal tulip]

I had a hard time trying to find a good course to start learning

[hollow sentinel]

whattt I wanna see

[misty flint]

andrew mo

[grave frost]

and there was another guy that roasts whitehatjr - I forgot his name

[hollow sentinel]

andrew Ng is the best stuff you'll find

[paper lake]

^^^^^

find him in coursera

[primal tulip]

And I completely feel the pain of all those bootcamps saying that in 2 weeks you'll get a 6 figure salary at the top companies.

[hollow sentinel]

but you need to know the math behind it

[primal tulip]

Yeah, I did his ML course.

[paper lake]

thats bs

[primal tulip]

But I have to redo it.

[grave frost]

lol, who does them anyway?

[primal tulip]

I like datacamp.com

[hollow sentinel]

these goddamn people who think they'll make 6 figures after 6 weeks?

[paper lake]

you dont get a 6 figure salary after learning in two weeks,,,,

[hollow sentinel]

it's just a psychology trick

[paper lake]

thats.... impossible

[hollow sentinel]

playing w peoples motivation

[grave frost]

bs more like

[hollow sentinel]

and all you are is a script kiddie anyways you just execute the code they do no questions asked

[primal tulip]

Doing that one, everything from Andrew at coursera.org and trying to get a cert at some cloud service. Most likely AWS

[grave frost]

but there are plenty of them around so apparently it seems to work; greed is a big motivator

[paper lake]

freecodecamp is also a good resource

[hollow sentinel]

I have seen so many tik tokers endorse these bootcamps and it makes me sad

[primal tulip]

Because they get paid lol

[hollow sentinel]

yeah

[paper lake]

tiktok is inherently bad though xD

[misty flint]

6 figures after 6 weeks. sounds like something a fake guru would say

[hollow sentinel]

there's some good people on there

not many

but some

[paper lake]

sadly some sounds so few

[grave frost]

I find vines on YT to be much more entertaining

https://www.youtube.com/watch?v=nX0NDmm1K1g

[paper lake]

sadly it is 11:50PM so goodnight

[grave frost]

adios

[misty flint]

gn friend

[misty flint]

"they just hate me" its the algorithmic bias

[grave frost]

pretty sure no one uses these captchas anymore

atleast the ones I see 🤷

most of them involve just ticking a box and going on

they use ML apparently to identify if the mouse movement is from a bot or a human, tho not much details have been released

[misty flint]

that means they need to collect mouse data

[hollow sentinel]

you know companies use ML to track what their employees do on their work laptops

they can track your key strokes and stuff

see what emails you sent

exactly what you're doing

[grave frost]

yeah, thats pretty common tho they dont use much ML - its just the plain old stuff

keyloggers and so on

you are advised not to use work laptops for personal stuff anyways

[hollow sentinel]

yes

[grave frost]

you won't know if they do implement it, most likely the box is false and they just analyze snippets of your mouse movements when your browse the site

rather than checking it before

[misty flint]

[grave frost]

IMO they would get more data if they analyze your browsing history 🤷

like if you are visiting several hyperlinks in small amount of time, that could indicate some automation (hmm..that does seem viable)

[misty flint]

google wont give up that valuable info easily

[grave frost]

recaptcha is owned by google

[yeet]

[misty flint]

[hollow sentinel]

guess I'm never gonna be free from big brother then huh

[grave frost]

just blow up your router and go to the jungle

[hollow sentinel]

also everything I use is Apple meaning Apple has all my data

also meaning they're probably selling it to third parties

[misty flint]

im okay if google is big brother. better than facebook. i dont trust them

[grave frost]

apple has the best privacy IMO

[misty flint]

[hollow sentinel]

according to who?

[grave frost]

they have the T2 security chip, and they don't allow as much freedom as app devs on apple store get in terms of permission

[primal tulip]

On a side note. There's a captcha solver that uses Google's own language audio AI engine to solve the audio part of the captcha.

Fight fire with fire

[grave frost]

it doesn't work that well

the discriminator always comes to be better, so you wont be able to generate a very good captcha

[primal tulip]

7/10 times it works. You can just refresh the web again if you want to automate stuff without human input.

I kind of do that at work lol.

[grave frost]

I believe there was some mathematical proof for that (I visualized it in some video) but that is the reason why they moved aways from puzzle based captchas

its one of the basic properties of GAN, and the reason in general why GANs work (not sure about the maths, so has to be confirmed by someone)

[grave frost]

plus if you fail more than once or twice, doesn't it temporarily ban you from attempting?

[primal tulip]

After you solve a captcha succesfully sometimes the very own engine asks you to train some more challenges. Not because what you did was wrong, but to properly use you as a manual classifier.

[grave frost]

also, isn't it illegal to collect captcha data?

and using it on google's own servers is kinda trolling them

[primal tulip]

Not quite. They don't ban you if you don't send like 1000 queries a second from the same IP. If you do less than 50 per sec, you'll be fine. You can always clear the cookies and get a new IP

[grave frost]

MAC?

you gotta change that also

[primal tulip]

No need for MAC.

Not for the captcha at least.

[grave frost]

in sites that allow you to use them for a month for specific times, changing IP+MAC does the trick (I had a script for that)

those were fun times

[primal tulip]

They check only the IP, the browser info, such as header, version and stuff like that. You could, but I don't ever change Mac addresses.

[grave frost]

cool

[remote fossil]

I'm trying to train an agent to in a racetrack environment but I am getting strange results when I plot the reward against episodes, if anyone can have a look at it, Id appreciate it

https://colab.research.google.com/drive/1NGEkVl1gJXk6OFkG6-X1DRbdHLUa4WXt#scrollTo=a8UaQGSYnajO

[arctic wedge]

Hey @remote fossil!

Uh-oh! It looks like your message got zapped by our spam filter. We currently don't allow .txt attachments, so here are some tips to help you travel safely:

• If you attempted to send a message longer than 2000 characters, try shortening your message to fit within the character limit or use a pasting service (see below)

• If you tried to show someone your code, you can use codeblocks
(run !code-blocks in #bot-commands for more information) or use a pasting service like:

https://paste.pythondiscord.com

[remote fossil]

not sure how I can share the txt file thats used to build the enviornment

[grave frost]

Gdrive it

[remote fossil]

https://drive.google.com/file/d/1r1pYaHiK8MzSDLqy0tkVFTTvlLcRoyF7/view?usp=sharing

[misty flint]

thats why

[primal tulip]

By the way, sorry if my rant is not appropiate for the channel. I just joined this server yesterday. How's the random channel called?

[hollow sentinel]

#ot0-psvm’s-eternal-disapproval

[primal tulip]

Thanks.

[misty flint]

its ok. captchas are kinda like ai

ai is a very broad field

tbh

[primal tulip]

Yeah, but I meant that since I wasn't asking or answering anything in particular. It was a bit of meaningless rant on my behalf.

[misty flint]

its ok

i found it interesting

[serene scaffold]

I'm trying to get more familiar with parts of the data science stack I haven't used yet. Right now I'm trying to preprocess the Titanic dataset and it seems the sklearn preprocessing tools were designed to simplify problems more complex than the one that I'm having

for example, I want to normalize the ages of the passengers by squishing them between 0 and 1, and convert the passenger class into a one-hot. And then the end-result would be array like [a, b, c, d, e] where only one of a, b or c is 1 (for the passenger class), d is float between 0 and 1 for the age, and e is 0 for men and 1 for women, or something.

[heavy tundra]

Do people typically run machine learning projects locally or in the cloud?

[primal tulip]

Why don't you try to wrangle the data in pandas? @serene scaffold

[heavy tundra]

I'm trying to train a model with YOLOv5 in Jupyter on my laptop and it took like 20 minutes to run 3 epochs

[primal tulip]

Depends on the size of the data and your hardware specs. At the cloud is usually faster but costs obviously more than running that in your old pc which takes longer. You can use Azure free trial since it's the longest one from the big ones I think (vs Google Cloud or AWS) but don't take my word for a fact.

[heavy tundra]

alright thanks

[serene scaffold]

I'm using pandas, yeah, but I want to learn more about sklearn.

And even though I gave a list earlier, it's going to be a pytorch tensor.

[primal tulip]

I'm not used to work with sklearn, but reading some of the documentation https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html
you have the .OneHotEncoder() with the .fit_transform and the .transform() methods. I would start there.

https://www.youtube.com/watch?v=irHhDMbw3xo
This guy drops the age column, but still does a one-hot encoding with the Titanic dataset.
@serene scaffold

[serene scaffold]

Thanks, I'll take a look at that video!

[noble sand]

Anyone know how you would extract classified Named Entities (NEs) from a NLTK tree? I'm having trouble grabbing these NEs and assigning it to a Python lists 😦 When I traverse through the NLTK Tree object, for some reasons no leaves are getting identified, for me to actually start retrieving the NEs

[grave frost]

cloud is much better. for beginners, colab is so much easier. though the best path is to invest in your own hardware, I recommend you ditch your laptop as soon as possible because YOLOv5 is pretty small anyway

[fresh tinsel]

Anybody know any good videos on Pandas Data Manipulation and Data Tidying?

[heavy tundra]

how long should training a model take on hardware

[grave frost]

which hardware

[heavy tundra]

nevermind, I guess it depends on the computer, model, images, etc.

[grave frost]

On laptop - weeks (due to constant thermal throttling)
On Colab/cloud - a couple hours usually

you can still fine-tune models on laptop, but that is pretty restrictive

[heavy tundra]

Does colab let me save the models locally?

so I could train it in the cloud and use it to detect something locally

[grave frost]

olab saves the model on google drive since it is the fastest and most reliable. after that, you can download the model from google drive to local

but if you have bad internet, then you should invest in building your own Deep Learning rig

[heavy tundra]

how does internet influence it

[grave frost]

its downloading, man

you download the model checkpoint and use it to do inference locally

[heavy tundra]

oh ok

I thought you meant my internet would influence how fast colab runs or something

[grave frost]

nah. for beginners, colab is the best. if you want extra power, you can pay for colab pro subscription

[heavy tundra]

alright thank you

[grave frost]

cool, no worries

[granite wolf]

please could someone help me with a groupby in pandas?

ObservationDate Country/Region Confirmed Deaths Recovered
0 01/22/2020 Mainland China 1.0 0.0 0.0
1 01/22/2020 Mainland China 14.0 0.0 0.0
2 01/22/2020 Mainland China 6.0 0.0 0.0
3 01/22/2020 Mainland China 1.0 0.0 0.0
4 01/22/2020 Mainland China 0.0 0.0 0.0
... ... ... ... ... ...
236012 02/27/2021 Ukraine 69504.0 1132.0 65049.0
236013 02/27/2021 Netherlands 16480.0 178.0 0.0
236014 02/27/2021 Mainland China 1321.0 1.0 1314.0
236015 02/27/2021 Ukraine 50582.0 834.0 44309.0
236016 02/27/2021 Netherlands 255335.0 3732.0 0.0

im trying to group by ObservationDate and Country/Region

summing confirmed, deaths and recovered

[serene scaffold]

Any reason you're using nltk and not spacy?

[serene scaffold]

it would be easier to experiment if you provide the data as comma separated values, though let me see if I can figure it out.

[granite wolf]

thanks, i think i've figured it out using: df2 = df.groupby(['Country/Region', 'ObservationDate']).agg({'Confirmed': 'sum', 'Deaths': 'sum', 'Recovered': 'sum'})

however it creates like new but lower columns that aren't counted as columns by pandas?

[serene scaffold]

what are lower columns?

!paste

[arctic wedge]

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pydis.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[serene scaffold]

^ I suggest providing the CSV there if you'd like any other kind of help with that data

[noble sand]

Nope, I don't really have any particular reasons for using NLTK tbh but compared to SpaCy, it does have a plethora of corpora and algorithms to choose from and also works with Stanford Tagger too 😉

[undone lotus]

hello

Does anybody understands how to import large csv file into a database table using python?

[exotic maple]

pandas read_csv?

how big are we talking?

i've handled up to 100 MB no probs

[grave frost]

https://www.youtube.com/watch?v=iEFqcMrszH0 Generating art ot music using a GAN

[small yarrow]

this is really cool, if there's ever a release of this package, I will definitely be trying it out

[misty flint]

is spacy much better than nltk? were thinking about using nltk for a telegram sentiment analysis project

[exotic maple]

oof. I was "trying" to test a few models with 100k x 1k array and PC is so hot I can cook a steak over it

[odd lion]

https://youtu.be/7uBNCN6v_gk?t=94

[exotic maple]

F CPU

[misty flint]

[grave frost]

shoulda used cloud

[serene scaffold]

I find have enough experience with nltk to say. What functionality do you want from either of them?

I've mostly used spacy to get token-level features

[exotic maple]

I finally got the shit model to converge

score -> 0.9.

[grave frost]

not bad

[exotic maple]

but now im scared of getting precision, recall for comparison ugh

im going to burn my shit HAHA

[grave frost]

CV?

[exotic maple]

I had to lower it to 2 CV for normal score

xD

i was trying default 5

[gray phoenix]

So I'm taking a MOOC for data analytics, I never took a stats class before. Every time a new stats term is mentioned, Im having to google what each word is.

Do you think I should cut my losses and take a stats course on Khan Academy to catch myself up?

[serene scaffold]

what stats terms?

[gray phoenix]

P Values, R squared, adj r squared, F test

These are a few, and im probably only a week in

@serene scaffold

[serene scaffold]

@gray phoenix is googling each new stats concept helping you keep up with the course?

[misty flint]

ah gotcha.

[serene scaffold]

also what is the name of the MOOC?

[gray phoenix]

@serene scaffold

Yes and no, I'm learning what it is. But I'm honestly concerned its going in one ear, and out the other.

Yes, in the sense that I am learning what it is.

[gray phoenix]

EDX, Predictive Analytics with Python.

Company is paying for it, so im not TOO concerned about "passing". But I am more interested in trying to learn and retain it

[misty flint]

@gray phoenix https://youtube.com/playlist?list=PLblh5JKOoLUK0FLuzwntyYI10UQFUhsY9

Check out Statquest. Very easy to digest videos

[gray phoenix]

@misty flint Thank you!

I'll take a look at this.

[misty flint]

he makes it fun with a little tune at the start of each video

np

i love statquest lol

[serene scaffold]

My thinking is, the advantage of taking a stats course to supplement the main course is that you'd learn the stats concepts in a structured way. But I don't think you can pause the main course while you finish the stats course.

[misty flint]

i have a stats midterm this week

[serene scaffold]

that's rough buddy

[misty flint]

its alright. this is literally my 5th time taking stats. all in different flavours

[serene scaffold]

that really sucks

[gray phoenix]

yeah thats exactly right. I was just thinking about taking the loss on it. I do keep the course. So I was going to revisit once I catch up in stats

[misty flint]

its alright. i think this last time, things are finally becoming internalized, instead of just memorizing formulas, yknow?

seeing bayes rule in ML was cool

[paper lake]

i watched statquest. highly recommend it

[misty flint]

[serene scaffold]

@gray phoenix do you need help understanding P values, for example?

[misty flint]

ive seen most of the stats vids. need to go through the ML ones

[gray phoenix]

I think right now for me atleast, I just gotta start to learn it. I'll definitely reach out if I have questions.

Thank you!

[misty flint]

best of luck

[wanton laurel]

I have some bank transaction data and one of the features is a short description
for example some values are "DEBIT - east coast rail" ; "2019-7-14 UBER" ; "Hotel Ritz" and so on. Struggling to find a way prepare this transaction description feature to predict the label of transaction ("Travel" "Accomodation").

[misty flint]

use nlp

[undone lotus]

10gb csv

[misty flint]

https://arxiv.org/abs/1701.06538

137 billion parameters

ofc jeff dean would be coauthor

[lapis sequoia]

Does anyone know cv2?
I really need help

[serene scaffold]

it's best to ask the question you have right away.

[lapis sequoia]

@serene scaffold I wanna live template match a video

how do i use numpy arrays in cv.matchTemplate

without downloading each frame

[serene scaffold]

I probably don't know how to help with your question, I'm just letting you know that asking your question is more likely to attract someone who knows how to help than stating the general topic of an unasked question.

[lapis sequoia]

:/

[prisma willow]

im trying to solve this using numpy/pandas/matplotlib and i dont know how to get 'r'squared ive tried youtube and google/khan academy and i cant find anything that uses a formula like this

any help to point me in the right direction would be appreciated

khan acadmey like others only ever seem to use y=mx+c

[serene scaffold]

are you having trouble with the definition of the "coefficient of determination" or how to encode the solution?

[prisma willow]

both

[shut slate]

Hey, anyone know anything about powerbi?

[serene scaffold]

Go ahead and ask the question you would have asked if someone said yes.

[shut slate]

You know how in a line graph there is a trend line that you can add in the analytics tab?

It doesnt show up for me

Like lol

[exotic maple]

@serene scaffold Coefficient of determination is, in its simplest form a definition of how much the indepedent variable(s) affect the dependent one

[serene scaffold]

Thanks, though A_new_player is the one who wants help with that

[exotic maple]

depending on the statistic, R2 can be computed directly (literally correlation squared)

or...you need to obtain via other means.

Linear regressions almost always generate R2 directly

[prisma willow]

ok but using i.e. matlab/numpy package how can i find the r squared?

i have no idea how to incorporate 't' data or how to sub anything into a/b/c

[exotic maple]

i'm going to assume that T is student's T score

If my memory serves me right students t is calculated just like a Z score

(value - mean) / std dev of the SAMPLE

o wait you werent asking how to calculate t lol

honestly, for the first, the wiki explanation is very indepth. https://en.wikipedia.org/wiki/Coefficient_of_determination

I really dont know what that T is thou

[prisma willow]

yeh but again, it doesnt seem to get me anywhere
https://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2

[prisma willow]

doesn't explain what the (model) is either

[exotic maple]

it says right there...below lol

next sentence

"where p is the number of explanatory variables"

[prisma willow]

which is?

[exotic maple]

uhm. Your features

all the Xs of the regression

[misty flint]

@prisma willow https://youtu.be/2AQKmw14mHM

[prisma willow]

isnt that just 'n'

[prisma willow]

thx ill take a look

[exotic maple]

n is data points

[misty flint]

do it. its really short

[exotic maple]

meaning, your samples

rows, etc

[misty flint]

will be v helpful

[paper lake]

trust rex

[prisma willow]

tell me what ''n' and explanatory variables are

[paper lake]

i trust him

[misty flint]

thanks

[prisma willow]

ur saying n(sample size) is the number of rows of 'x's and so are the explantory variables?

[exotic maple]

I recommend you watch the video. I'm kinda tired and i don want to mislead you with a shitty explanation

also, you shouldnt be watching that wiki section
. Your problem EXPLICITLY states UNADJUSTED R2. you're reading adjusted R2

[misty flint]

oh rip

unadjusted R2, so calculate it by hand or..?

[exotic maple]

the problem he has some t values idk wtf they are lol

i have never related students t to R2

[misty flint]

maybe just a random variable

[exotic maple]

im sure there is a way though, because some linear regerssions assume Gaussian distribution

[exotic maple]

maybe. someone should tell that teacher to not recycle important variables though...

t, z, those letters have meaning in statistics lol

[misty flint]

ye thats poor practice

tsk tsk

[prisma willow]

using python

• packages

also video helped with some logic, but again it doesnt help with the question itself

[ripe forge]

What's the Question?

[prisma willow]

im trying to solve this using numpy/pandas/matplotlib and i dont know how to get 'r'squared ive tried youtube and google/khan academy and i cant find anything that uses a formula like this

any help to point me in the right direction would be appreciated
khan acadmey like others only ever seem to use y=mx+c

if i could get 'y' and 'f' i could do it
https://www.geeksforgeeks.org/python-coefficient-of-determination-r2-score/

[ripe forge]

Hm. What's t?

[misty flint]

do you have to use matplotlib? i can never get it to do what i need it too lol

its trying a polynomial function instead of a linear one

[prisma willow]

im trying this
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

[prisma willow]

dont know what 't' is

thats all thats in the question

[ripe forge]

T must stand for "true values" I think.

[misty flint]

oh nice. it looks like it returns what youre looking for R^2. just gotta use the score() function

[ripe forge]

I guess it doesn't matter whether t is true or predicted does it

[prisma willow]

@misty flint
i dont know for this line:
X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
should the second values in each list be the 't'?

[prisma willow]

sure?

[ripe forge]

No I'm not sure. More like speculating

[misty flint]

possibly. then you could use the functions below maybe

im not too sure tho since i have a hard time interpreting what its asking

[prisma willow]

assuming that that's right then this is the only confusing line with what to sub for...
y = np.dot(X, np.array([1, 2])) + 3

[misty flint]

oh you have to change that one completely

thats the function

in that example it looks like its doing the dot product of X * [1,2] then adding 3 bc why not

[prisma willow]

then it comes right back to the initial question
what is y=ax^3 + bx^2 etc.... whats 'a' and 'b' and 'c'?

[misty flint]

that is literally the part when i was like this

idk if those coefficients are what you plug in t in for or what

no idea

[oak elk]

I have to Draw a contoured field plot to illustrate the particular pollutant anomaly variations in terms of the month (y-axis) and year (x-axis)

I am not able to get months on y axis

Can anyone help?

[cedar nacelle]

@oak elk can yo please share your code which contain graph algo

[misty flint]

in statistics, how come they always say its impossible to determine causation for something?

yes, i get that correlation does not equal causation but at this rate, can you EVER prove causation

without being omnipotent

[misty flint]

literally me

[astral path]

heh e

ok serious question time

how do you determine where to set the x bounds for a plot?

I'm trying to visualize the distribution of a variable

but it's kind of heavily skewed because there's some outliers

i.e. there's only 10 occurances of a plays-over-expected greater than 6 in a list of 7000+ entires

[misty flint]

your data always has wild outliers

[astral path]

yeah loll

it's much more readable when i set the x bounds to 0 and 6

[misty flint]

maybe one with and one without outliers then

[astral path]

but still not that great

is there a good way to determin it?

[misty flint]

also you know what distribution this looks like

[astral path]

what?

[misty flint]

maybe log normal

[astral path]

huhhhhh i guess it does look like that

plays over expected is basically just a measure of how many more plays I have for a song than the average user has

also

what exactly might the purpose of having gradient coloring like this be? (not my plot)

[lean ledge]

It's not always impossible

If you know there's no confounding variables, you can draw causational networks and use your knowledge of the system to infer the causation

At least to an extent

[high badge]

does anyone know how to access kerastensor values like a vanilla tensor?

[misty flint]

ohhh. wait then that means your outliers mean something. those 10 songs are very popular, most likely outside standard distribution too. but obv you can remove them if youre trying to analyze just the bulk of data

[lean ledge]

Probably not much more than "it looks nicer"

(for that particular graph)

[astral path]

yeah my graph's purpose is pretty similar so i think i can just do the same if i choose

[astral path]

yeah yeah like i'm trying to make a plot that will show me what the general trend is for plays above expected

wait

i need to take into consideration number of plays

[misty flint]

the only thing i can interpret is that at each timestamp, you have multiple values, so the coloring helps you distinguish some of the more granular data but idk why they dont just divide the data up instead

[astral path]

ohhh yeah that would make sense

[misty flint]

obv depends on the use case

[astral path]

i could always use hue for weight but that wouldn't be too helpful for some cases which don't have a lot of plays and aren't very visible

man i need to take a data viz course so much

too bad all the ones at my uni have heavy prereqs :(

[misty flint]

maybe take a look at a few books?

also i found a cool library that does bayesian causal networks

https://causalnex.readthedocs.io/en/latest/04_user_guide/04_user_guide.html

[paper lake]

oooh thanks

[astral path]

i ended up doing a parallel plot for the purposes i was wanting

O LMAO

i didnt realize that recorded my audio too

[paper lake]

https://github.com/ossu/bioinformatics dunno if this is a good resource for anyone interested in bioinformatics

[marsh onyx]

Is it possible to import graphs?

[serene scaffold]

import a graph instance? when you import a Python module into another module, you get all the variables defined in the global scope for that module.

[marsh onyx]

@serene scaffold ok?

[serene scaffold]

did I not understand your question?

[marsh onyx]

well u did

but i ddnt

[serene scaffold]

see #❓｜how-to-get-help to open a help channel and ping me when you do 👍🏻

[misty flint]

[paper lake]

frik u are too addicted of that sticka :((

[tough geode]

can I ask question anyone in here do the Shopee Code League - Multi-Channel Contacts

[lapis sequoia]

https://www.investopedia.com/thmb/z3VbR2evm2dUlXvPWYesjWxxSHI=/3274x0/filters:no_upscale():max_bytes(150000):strip_icc():format(webp)/InvertedYieldCurve2-d9c2792ee73047e0980f238d065630b8.png

how is this curve named?

[misty flint]

i wanted to recommend books but i havent read any good ones about data viz principles. all i have are podcasts

[lapis sequoia]

*graph

[misty flint]

[paper lake]

sadsuu

[misty flint]

well heres one about florence nightingale and how she used data viz to get she what she wanted

https://open.spotify.com/episode/4cPEJLaOuG7de0Ei860yZj?si=tnFCl_NaRXGtcJcC5kV7lA

[lone drum]

How I can install geopnadas

I am facing this error

[paper lake]

@lone drum https://geopandas.org/getting_started/install.html#dependencies

[hollow sentinel]

http://blaqueyard.com/download/Python Data Visualization Cookbook.pdf

this is a pretty concise one that I found

[half ivy]

In pandas can I select a column with '.' - as shown in the highlight below ".sales"?

[hollow sentinel]

https://pandas.pydata.org/docs/getting_started/intro_tutorials/03_subset_data.html

try that

[lone drum]

@paper lake not worked

I tried installing all dependencies

[paper lake]

@lone drum the error said there that u should have that "GDAL API" or something installed

i aint familiar with GeoPandas

[lone drum]

Ok np

[hollow sentinel]

if you're more of a videos guy you can use this too https://youtu.be/xvpNA7bC8cs

this guy is goated

I was watching his vids all the time when I was using Pandas

[half ivy]

Awesome. Thanks so much

[hollow sentinel]

yep

[lapis sequoia]

how do I motion track colors in opencv??

[hollow sentinel]

https://github.com/CalvinDSeamons/Motion-ColorTracking

[misty flint]

gpus are faster on matrix operations than cpu

this is why training a model on a gpu tends to be faster

[hollow sentinel]

Is it bad that today I learned that a GPU is a graphic processing unit

[misty flint]

no

[hollow sentinel]

am noob

[misty flint]

before ML, gpus were mainly big in the gamer community

[hollow sentinel]

interesting

[exotic maple]

just dont use your gaming PC thinking you can handle ML

-cries in former graphic cards-

[misty flint]

[paper lake]

:3

[misty flint]

rip

🕯️

[exotic maple]

killed my 970 lel

bought a 1070

[paper lake]

killed it again

[misty flint]

[exotic maple]

nah thats fine. 1070 is a beast

970 was old as well

[misty flint]

[exotic maple]

but holy it popped like popcorn

[old thorn]

will a rx 570 in a desktop perform better when training CNNs rather than my macbook?

[exotic maple]

at that point just use cloud

[misty flint]

cloud has gpu options too

[exotic maple]

i havent used cloud cause im a lazy fuck

[misty flint]

if youre a student, they give you free credits

[old thorn]

well its not that, ima gamer and coder so im building a gaming PC, on my mac I use Colab, but since Im building a PC I wanna know if its worth going local rather than cloud

[misty flint]

how much ML are you doing

[old thorn]

casual

[misty flint]

nah just use the cloud then

[old thorn]

i only go hard core for competitions

kk ty

[misty flint]

build your stuff for gaming

sounds good

[paper lake]

gpus are for gaming!! abooos google colab gpus for freeee

[misty flint]

[old thorn]

lol yea a mentor of mine told me he built a desktop with a dedicated GPU for deep learning

[exotic maple]

sounds moronic.

Deep learning on a desktop is kind off dumb lol

[misty flint]

itll still take forever to train

[exotic maple]

i can barely manage a few random forest or GDtrees on my pc xd

[old thorn]

ah ok i see

[misty flint]

on the cloud, you can just run multiple gpus rather than just running 1 for like a week

[misty flint]

also less expensive

[paper lake]

maybe it is a desktop with a lot of gpus

[misty flint]

[exotic maple]

ive been running that crap for like 1 hour xd

its a GBDT

my comment is big brain thou

[misty flint]

hey i just learned about grid searches

[old thorn]

nah hes a 1st year at stanford he cant afford SLI or Crossfire after tuition Lul

especially in this market

[exotic maple]

RIP chip produces

producers

nvidia be like:

[old thorn]

lol

[misty flint]

quite literally

nvidia is like: good thing ML took off.

[paper lake]

Silicon shortage be like

[misty flint]

dont they make specific chips for ML now

[paper lake]

oooof

[misty flint]

[exotic maple]

I only saw chips for mining

which i htink its the greatest waaste of power in recent history

[misty flint]

hmm i dont remember if i heard it correctly on a podcast

i think thats what they said

[paper lake]

AMD too is following nvidia. also planned to create gpu for mining

xD

[misty flint]

[paper lake]

always

[exotic maple]

I'm going to sound heretic here, but is anyone else interested in ML / AI here, but to a simple applications perspective?

[paper lake]

imagine mining can use more power than a whole city? freaking helll

[exotic maple]

As in, limited uses, small implementations, quick wins, etc

I'm liking everything i'm learning but i dont see myself working on massive DL or AI stuff lol

[misty flint]

https://developer.nvidia.com/tensor-cores

[misty flint]

yes. low hanging fruit

thats what i like my projects to be

[exotic maple]

is that ironically or an insult or real? xd

I do want the low hanging fruit lol

[misty flint]

real

i see of it as another tool to use

[exotic maple]

This.

[misty flint]

[paper lake]

btw we are planning to create a package in julia related to epidemiology or protein folding. is it okay to ask advice from u? just not on julia stuff

[exotic maple]

I literally was telling that to my CS friend

[misty flint]

[exotic maple]

like

he's into DL and all that

and i'm like: "cool dude, thats why you studied CS and I didnt lol"

[misty flint]

sure. you can ask general questions. i cant promise ill have answers

[paper lake]

yeah sure thanks :3

[misty flint]

[exotic maple]

Julia looks sexy but i havent found a single piece of deent learning material

[misty flint]

lol idk if i can do a pure CS degree. im in grad school now but for AI

coming from a non-technical field

[paper lake]

pffft

[exotic maple]

I'm coming from industrial engineering

[paper lake]

im dead

[exotic maple]

most people in my field and in most admin feels find their epoch in excel lmao

[paper lake]

freak i am an undergrad in microbiology :((

[exotic maple]

but then : WILD 30MB DATA FILE SHOWS UP

[misty flint]

[paper lake]

and excel cant eat that up xD

[misty flint]

wait is that really the limit of excel?

[exotic maple]

depends

[misty flint]

idk why i thought it was bigger

[exotic maple]

ive never seen excel opening anything above 25 MB lol

[misty flint]

all i know is they cant do more than 1mil rows

[paper lake]

depends yes, there is a max row and col in excel

but idk how large

[misty flint]

starts to break

[exotic maple]

excel breaks at around 100k rows

[paper lake]

i remembered the UK have large backlogs of data because they used excel for covid

[misty flint]

[exotic maple]

wut

the UK?

they have a data science ecosystem there

[paper lake]

not sure

[exotic maple]

they couldnt get anone to pandas it?

[paper lake]

but there was large backlogs because they used excel

[paper lake]

yeah i think i have read it from last year

UK got hit so bad by covid iirc

[misty flint]

i saw a job listing for an internship at a local power company to "migrate all their data from excel to 'python' to run analyses"

i literally died

💀

[paper lake]

[exotic maple]

PLOP

imagine getting hired to do that

you'll end up building their whole data infraestructure

[misty flint]

imagine your bosses not knowing enough to even see if youre doing your work properly

migrate to 'python'

i still cant get over that

💀

[exotic maple]

eh

all my career LMAO

[paper lake]

hahahahhahaa.... i can only laugh in excel

[exotic maple]

good side, my bosses never annoy me

bad side... figure shit out yourself lol

I, without knowing crap about dbs, implemented the most disgusting, heretical and repulsive postgresql server you can imagine

[misty flint]

ive heard that you never want to be the first data scientist at a company bc youll end up building the whole data infrastructure like you said

then itll be a pain to fix everything later

and every data scientist later on will hate you

[paper lake]

and then they will say your infrastructure is shit not knowing your struggles of being alone :((

[misty flint]

sadoru

[exotic maple]

lol

wtf

[paper lake]

imma watch Redo of Healer tomorrow. goodnight

[exotic maple]

I made a postgre to keep track of my weight and i havent update in months

[exotic maple]

bro. the ending of last chapter AHAHAHA

such a fucking meme

thank you, gridsearch. after 2 hours of burning my pc finally got somethng :v

[misty flint]

my prof has been lecturing about error backpropagation for a while and im still like

hes one of those profs thats all rambly

[astral path]

Is there a good way to compute correlation between location (as in on a map) and a numerical variable?

I'm trying to figure out if there's a correlation between location of artist who I listen to and # of plays they have

[exotic maple]

you need to convert location to an encoded variable

a location is a nominal value so you cant numerically process it.

[astral path]

ahhhh ok

[exotic maple]

even if you have LAT/LON of location, i dont think it would work

[astral path]

yeah that's what i'm working with

[exotic maple]

lat/lon is still a type of categorical variable, its a coordinate. -40 E has no numerical value in itself

[astral path]

that makes sense

thanks for clearing that up, i was about to make a pretty big mistake i think

[exotic maple]

I had a similar issue not long ago

lat lons are given as numbers

but in reality they are coordinates, nominal data, not a continous value in itself

[astral path]

I found a pretty good model called Moran's I but I'm not sure it would work here

how would you encode them?

[exotic maple]

how many unique pairs you have?

you could use sklearns OneHotEncode

[astral path]

I don't know for a fact but I think it's anywhere from 500-1000

[exotic maple]

onehotencode turns basically every unique category into a binary variable

so, that would create 1000 features from 1000 unique pairs of lat lon

[astral path]

ohh ok that makes sense

wait 1000 unique values for one variable or 1000 categorical variables?

[exotic maple]

1000 categorical

[astral path]

ohhh

wouldn't that be slow?

[misty flint]

like zipcode

[astral path]

wait lol yeah

[misty flint]

OrdinalEncoder is a decent one too if you dont want that many categories

[exotic maple]

its what you are asking for lol

[remote fossil]

My reinforcement learning agent takes about 5 seconds to train over 150 episodes, however, when I implement a function to average 20 agents it takes over 3 minutes, basic math says it should take 1 minute 30, any ideas?

[misty flint]

function probably isnt linear

something something big o notation

something something

also other things could affect training time

[remote fossil]

def agent_avrg():
       
    #initialise average undiscounted return among x agents
    #episode number : return value
    avrg_undiscounted_return = {}
    for itr in range(150):
        avrg_undiscounted_return[itr]=0
        
    env = RacetrackEnv()
    for i in range(20):
        env.reset()
        a = Agent(env, 0.2, 0.9, 0.15, 150)
        undiscounted_return = a.sarsa()
        
        for ep in avrg_undiscounted_return:
            avrg_undiscounted_return[ep]+= undiscounted_return[ep]
    
    for ep in avrg_undiscounted_return:
        avrg_undiscounted_return[ep] = avrg_undiscounted_return[ep]/20
                    
    return avrg_undiscounted_return

[remote fossil]

ah makes sense, the above function is my averaging function, do you see a way to improve speed/simplify it

[shut slate]

Why does this not load 😦

[serene scaffold]

@shut slate Please share this text as text

!paste

[arctic wedge]

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pydis.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[shut slate]

#Loading the dataframe
df = pd.read_csv(r'C:\Users\polsm\Desktop\pd.csv')
df

like it says it cant find the file? Am i getting the wrong directory or something

[serene scaffold]

Can you paste the entire error message into the paste bin linked to above, and then give us the link to the paste bin?

[shut slate]

https://paste.pythondiscord.com/eroquzaxut.sql

[serene scaffold]

where is your jupyter environment?

[shut slate]

localhost idk

I never changed anything

[serene scaffold]

I'm not a fan of jupyter for this reason

I would first go to the directory where that file is located and do python -c "import pandas as pd; pd.read_csv('pd.csv')", just to ascertain that there's not some weirder issue at play here.

[shut slate]

can I just upload the file somewhere and then use URL? worked for me b4 but i need never uploaded the file anywhere, just straight from url

now i get this...

https://paste.pythondiscord.com/asitodufig.sql

[serene scaffold]

the csv file needs to be on the same computer as the Python interpreter. Is this a Jupyter notebook on your computer or on a cloud platform?

[shut slate]

computer http://localhost:8888/notebooks/Desktop/Untitled4.ipynb?kernel_name=python3

is my url

I fixed it with a band aid

I uploaded the file to website and copy pasted the URL

#Loading the dataframe
df = pd.read_csv(r'C:\Users\polsm\Desktop\pd.csv')
df

Now it magically worked???

idk whatever

lol

[serene scaffold]

🤷🏻‍♂️

[shut slate]

thanks. I am sure I will be back

lol

[serene scaffold]

@shut slate what level of Python experience would you say that you have?

[shut slate]

complete noob lol

[serene scaffold]

I would encourage you not to use jupyter at all, then

It makes it look easier, but this is deception. it actually adds a lot of extra considerations and makes debugging more difficult.

[shut slate]

Yeah I noticed but my uni is kind of forcing us to use it

[hollow sentinel]

jupyter notebook is annoying with the cells

[astral path]

oh hey i recognize you from the #algos-and-data-structs channel

[hollow sentinel]

yeah I frequent this channel sometimes bc I wanna go into data science

it's a dream

[astral path]

I'm a data science/AI major rn

[hollow sentinel]

nice

[serene scaffold]

one way you can simplify the debugging process is to always run the cells in order, from top to bottom.

[hollow sentinel]

god knows how I'm gonna learn all the math behind this field

[shut slate]

Yeah I do that

How do you change the the column names from a new list you have?

[astral path]

is it a dataframe already?

[shut slate]

yeah

[astral path]

df.columns = ['col1', 'col2', ...]

[shut slate]

ok thanks

But let's say I defined the list already newColumns = ['Hospital', 'Provider ID', 'State', 'Period', 'Claim Type', 'Avg Spending Hospital', 'Avg Spending State', 'Avg Spending Nation', 'Percent Spending Hospital', 'Percent Spending State', 'Percent Spending Nation']

When i do df.columns(newColumns) it doesnt work