data-science-and-ml

[sharp jewel]

come guys i can't open file image in visual studio code PhotoImage("Logo.pnh")

[olive stone]

Hey, why am I getting this when importing Tensorflow?

2023-03-16 08:00:40.027703: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-16 08:00:40.664401: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/habbas11/miniconda3/envs/tf/lib/
2023-03-16 08:00:40.666338: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/habbas11/miniconda3/envs/tf/lib/
2023-03-16 08:00:40.666351: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
``

[wooden sail]

the way the polynomial is printed has nothing to do with how the object is stored internally. one can define how things are printed for any class, and they just chose this weird 2-line format for poly1d. you can evaluate the poly at any point

!e

import numpy as np
p = np.poly1d([1,2,3])
print(p)
print(p(1))

[arctic wedge]

@wooden sail :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 |    2
002 | 1 x + 2 x + 3
003 | 6

[wooden sail]

my best guess is that it's the number of convolution kernels applied

[violet gull]

how do you apply 0.7 kernals

thats the only way it goes from 384 to 256

[wooden sail]

no

you apply each kernel to all of the previous output

this requires an assumption on what you do about the extra dimension

iirc pytorch and tf average by default

e.g. you apply a 2D convolution to a color image, which has 3 layers

[violet gull]

ok how many different kernals does this layer have

[wooden sail]

if you apply a single filter, the output will have ony one slice

[wooden sail]

384, i would say

[violet gull]

so it goes from 1 image to 384

[wooden sail]

yes

[violet gull]

and the next layer

384 images to 384**2?

[wooden sail]

no

the output again has 384 slices, because we apply a new set of 384 filters whose results are averaged over the slices of the input

[violet gull]

what

[wooden sail]

say we have an image of size N x M x L

[violet gull]

thats a cube

how cube be image

[wooden sail]

dude all RGB images are cubes to start with

[violet gull]

assume grey scale

[wooden sail]

no, alexnet uses 3 channels in the input

[violet gull]

my brain doesnt work fast enough to use 3 channels

only 1

[wooden sail]

this is exactly the reason you aren't understanding what'S going on

the explanation NEEDS more than 1 channel

[violet gull]

im trying to recreate alex net but with 1 channel

[wooden sail]

you run into the SAME problem 1 layer later lol

even if we start with an image of size N x M

we then apply K filters of size 3x3, for example

the output of this process is of size N-2 x M-2 x K

[violet gull]

yes

[wooden sail]

so now we're again at a 3D "image"

[violet gull]

that makes sense

[violet gull]

my brain thinks of that as a stack of images

[wooden sail]

well, RGB is a stack of 3 images then. one red, one green, one blue

not so difficult

but that's irrelevant

[violet gull]

ok

[wooden sail]

now we go to the next layer

we have something of shape N x M x L at the input of the next layer

we apply K filters

now, the filters are 2D, but the input is 3D

this operation is not uniquely defined

what TF and pytorch do by default is apply the 2D filter to each of the L images in the input, then average that

so each filter yields a single slice of size N-2 x M-2 x 1

[violet gull]

what do u mean by average it

[wooden sail]

by averaging over the L results

i mean average

[violet gull]

average of what

[wooden sail]

of the L results

[violet gull]

specifically

[wooden sail]

we have L images of size N x M at the input

if i apply a 3x3 filter to each of the L slices, i get L images of size N-2 x M-2

we then average these L images

pixelwise

[violet gull]

so 384 becomes 1 image

which is an average of the 384

and that happens 384 times

[wooden sail]

yes

[violet gull]

and this one, we average 256, 384 stacks

[wooden sail]

no

[violet gull]

jesus that is a lot of computations

[wooden sail]

backwards

the input has 384 slices, the output has 256

so we apply each of the 256 filters to the 384 input slices, and average those 384 slices to get the result of the filter

[violet gull]

isnt that what i said

[wooden sail]

what you said reads backwards to me

[violet gull]

ok so its 256 * 384 individual convolutions

[wooden sail]

yes

[violet gull]

how the hell can a computer handle that

[wooden sail]

well, machine learning is slow

also convolutions can be done surprisingly efficiently

[violet gull]

mine run pretty slow

[wooden sail]

that's because you wrote your own stuff instead of using existing libraries

the reason it can be done quickly is that the operation can be parallelized

BLAS libraries use SIMD or CUDA to do this quickly in parallel

[violet gull]

i tried using BLAS and SIMD and it made it run slower

[wooden sail]

then you did it wrong 😛

[violet gull]

ok so for paraleleization

am i running every convolution on a seperate thread

or am i breaking up each convolution onto several threads

like 10 convolutions = 10 threads or 1 convolution = 10 threads

rust c# or java for rocM

[wooden sail]

both

you can do each conv in a separate thread, and have each thread use simd

[violet gull]

is simd like a library or an algorithm

[wooden sail]

simd are special instructions to the cpu

https://en.wikipedia.org/wiki/Single_instruction,_multiple_data

[violet gull]

so how do i actually write code that does it

[wooden sail]

you use an API for it

https://vksegfault.github.io/posts/java-simd/ here's an example

[violet gull]

u have experience in rust?

[wooden sail]

nope

maybe ionite does

[violet gull]

i have it all written in java rn

but java is old so im not against porting it

i also hate apache

alright ty Edd for the help ❤️

[fallow frost]

How should I go about saving data to parquet without loading everything into a DataFrame? (I already have pandas and dask as project dependencies so I would prefer to use one of those)

[coral bloom]

hey, would anyone wanna learn collobarate to learn neural networks from scratch without too deep of textbooks

I mean you need to know what neural networks are, python, how brain does it and lets figure it out from scratch

does anyone know how to make dynamic hidden layers?

anyone?

@violet gull bro, can you help?

[violet gull]

what

[coral bloom]

dynamic hidden layers

[violet gull]

what does that even mean

[coral bloom]

as in I want my neural network to recognize everything without me coding in each hidden layer

[violet gull]

thats not really how they work

[coral bloom]

ummm....can you tell me how they work?

[violet gull]

models are specifically designed for problem tasks

there isnt a one fit all

[coral bloom]

but gpt-4 is able to read every image right...

without any speccific details

[violet gull]

it can attempt it

[coral bloom]

which means it is dynamically making hidden layers and rearranging the weights

[violet gull]

no

no its not

it has a massive multi billion parameter model

nothing about the model is changing dynamically

[coral bloom]

and these parameters are not manually inputted right?

dynamically learnt...

[violet gull]

the parameters are the magic in the model

[coral bloom]

exactly ..

[coral bloom]

so...this right?

[violet gull]

correct

[coral bloom]

or wrong..

[violet gull]

they are not manually input

[coral bloom]

so, the network is able to dynamically create its own parameters when something new needs to be learnt right?

[violet gull]

no

the parameters are set in stone

nothing about them is changing except when the actual gpt developers are training the model

you showing the AI an image does not change the parameters

the parameters are static

[coral bloom]

mmmm, sorry if I'm super noob and say it this way

[violet gull]

np i enjoy explaining

[coral bloom]

lets assume there is a baby, he has a big neural network and needs to learn numbers

I saw vids on youtube how they teach them numbers and they show each stroke of the number to the baby

[violet gull]

can the baby do multivariable vector calculus

[coral bloom]

nope iss a smol baby

[coral bloom]

Now, I assume that their neural network dynamically creates parameters when the teacher says number one has a single line stroke and draws it multiple times

and so for 2,3,4,5,6 etc etc

So I assumed similarly, for a neural network, there must be a way to dynamically make these parameters by themselves right...

no?

[violet gull]

not from my research

[coral bloom]

mmmm, so such a neural network must have parameters which filter out strokes?

[violet gull]

the only dynamic thing ive seen is allowing the network to expand itself when it is struggling to train

[coral bloom]

expand itself means..?

[violet gull]

add another layer

increase the number of trainable parameters

[coral bloom]

YES, so it can add another layer to itself?

[violet gull]

technically yes but thats kinda useless

[coral bloom]

and it does it by itself right..?

[violet gull]

it can

[coral bloom]

so how does it do that?

[wooden sail]

you could do that, but afaik chatgpt doesn'T do that

[violet gull]

yuh this

[coral bloom]

Umm, but it is able to take in new information and learn right..

[wooden sail]

no

[violet gull]

my guess would be if the loss/epoch delta is too low it would increase the num parameters then retrain

[wooden sail]

i mean, in theory yes, but they have set it up so that it doesn't

[coral bloom]

GPT 4 demo showed how it read and understood the documentation

[violet gull]

no matter what anyone shows GPT 4, the model remains static

[wooden sail]

that's how all language models work

you show it something, and then it produces text that follows from the sample with high probability

the once the session is over, this is all lost

that's not learning, that's its inference process

[coral bloom]

mmmm, so let me get this: To make a neural network which is able to decode numbers, I need to manually make hidden layers to detect each pixel layer row? and detect the strokes?

[violet gull]

yes

[wooden sail]

if you wanna pass the numbers as pixel maps, yes

[coral bloom]

so, the same network cant be dynamically trained to exxpand itself and detect alphabets and stuff?

[wooden sail]

you could, why not?

[coral bloom]

because the parameters need to be coded manually..

[wooden sail]

they don't

[coral bloom]

or...do you think babies have an api which creates these parameters when we show them...

[coral bloom]

??

[wooden sail]

why are we talking about babies

[coral bloom]

cause...I am a noob and think raw neural networks represent babies and how they learn stuff

......and how they learn numbers

[wooden sail]

that'S a dated analogy and is making it more difficult for you to understand what is happening

[coral bloom]

so,```

1. A neural network of 540*640 inputs. Takes in images.
2. It needs a dynamically expanding output layer and dynamically expanding hidden layers (Parameters)````

[violet gull]

the best way i understood it was thinking of it as a big math equation
ax^3 + bx^2 + cx + d
where a, b, c, d are constants that the model is able to change to change the shape of the equation

[coral bloom]

mmm, true ig

[coral bloom]

so this is possible right? @violet gull

[violet gull]

its possible but it is not practical

be like trying to hammer a nail with a sponge

[coral bloom]

but it needs to transform into a plate when hitting a nail...

thats what GAI is right

[violet gull]

what

[coral bloom]

I mean if GPT-4 can do it;

[coral bloom]

General AI ...

[violet gull]

gpt does not do that

[coral bloom]

it sees images...it decodes them easily...it can read and say stuff

[violet gull]

yes

[coral bloom]

which means it is general AI

...no?

[violet gull]

"general AI" doesnt really have any meaning

[coral bloom]

It is the opposite of narrow AI, which can only perform a specific task

GPT is able to perform a wide range of human tasks

so general

[violet gull]

technically gpt is still narrow ai

it just is large

if your end goal is re creating gpt it aint gonna happen

[coral bloom]

:[

why?

I think gpt works because it follows babies learning methods

and thats why it doesnt sound stipid

[violet gull]

gpt is not a baby

[coral bloom]

yup its grown into an adult

but it was once a baby

[violet gull]

i think what you are thinking of is reinforcement learning

where it learns based on what it sees

gpt is not that

[coral bloom]

Okay OKay. So back to topic. Can you please tell me how to do this?

[violet gull]

go learn multivariable calculus

start with that

[coral bloom]

I did, I know multivariable calculus

And solved a shet load of them in engineering

and passed somehow 🙂

[violet gull]

mk

build a dense layer

activation functions

[coral bloom]

dense layer?

[violet gull]

backpropagation

[violet gull]

that is the very basic implementation of what gpt is built on

[coral bloom]

mmmm,

I understand that, but for now, I just want to make a function and api, where the network takes in images and I teach it various strokes

and whenever a new image is inserted, it can decode if its a number or a letter and what type it is. And is able to learn new letters

Teaching here does not mean coding each parameter

[violet gull]

use pytorch

it will do all the work for u

[coral bloom]

nu...I want to code my own neural network...

at least for the first time...

[violet gull]

ok so build a dense layer

whats the problem

[coral bloom]

eli5

dense layer

[violet gull]

?

[coral bloom]

like, I want to understand what dense layer means? I understand, input layer, output layer and hidden layer where classification of the input happens

and multiple hidden layers can be there to classify the input to precision. More hidden layers, more classification

so where does dense layer fit in?

and what is it?

[violet gull]

google it

its a big subject

[wooden sail]

GPT is NOT a general AI, and the creators have said so themselves

it's a language model

btw, there's something called the "no free lunch theorem" that tells us all estimators perform the same when averaged over all the possible data

this means if an estimator is good at one thing, it is necessarily bad at others

[coral bloom]

The creators don't know the monster they created...

I wrote a stupid paper on it 7 years ago..

and it was right on spot, cause, it shows what is general intelligence

[coral bloom]

trying

[wooden sail]

nope

[coral bloom]

https://docs.google.com/document/d/0B6hyCZyXH085TE92UzVDaWFrWW8/edit?usp=drivesdk&ouid=111815649746439647725&resourcekey=0-MtR8kEx1dP5SZ4A1SRqlsw&rtpof=true&sd=true

I wrote it

ig, thats wrong in many ways, idk but yup. Ill learn. Dense layer

dense layer

[wooden sail]

none of your references are relevant

[coral bloom]

Ill stop with the emos

but yup,

Thats what I think

lets see

imma try 🙂 Arrrr

[wooden sail]

i'd say the best way of understanding AI is to learn the maths

it is, after all, all maths all the way down

[hasty mountain]

What is this supposed to be? A college presentation?

[coral bloom]

ummm..nope, it was supposed to be a "paper" I wrote

[hasty mountain]

Bruh...

[coral bloom]

after deep thinking. Hry I ewas in 10th grade then so its alright 🙂

[hasty mountain]

Oh, okay then

[coral bloom]

in hinsight it did come true

to an extent

[coral bloom]

but do lemme know your thoughts 🙂

[wooden sail]

"deep thinking" is not the only thing that goes into papers

you need to back up everything you say

either from previous sources or by showing it yourself

[hasty mountain]

Oh, there's no thoughts. I was just going to suggest some books on neuroanatomy and neurology, to enhance the definitions...

And math is also important...

with non-linearity...

[coral bloom]

yup, true, but wats the flaw in the paper logic

[wooden sail]

i didn't read it, i saw the intro and the references and decided anything in it isn't backed up anyway

[coral bloom]

dang

[hasty mountain]

Also...when referring to humans in experiments, I wouldn't recommend using the term "subject", but "volunteers"

[coral bloom]

sir yes sir 🙂

[wooden sail]

this is how i always read papers btw. gotta quickly weed out the ones that you shouldn't spend time on

[coral bloom]

But I think youre missinng out on aa lot of content

and I sgree to weeding out stuff when you

[wooden sail]

content with no supporting evidence or proof

i only have so much time

[hasty mountain]

(Also brotip: if you ever want to make a research and a paper...avoid experiments with human volunteers...they're such a bureaucratic mess...)

[coral bloom]

need to read a shet load of papers professionally

[coral bloom]

agreed Yup

[wooden sail]

i'll never finish my phd if i go through every single paper

[coral bloom]

think of it more like a fun blog 🙂

[wooden sail]

even worse, i don't read blogs cuz they usually have super bad quality control and fact checking

unless it's a verified source

[hasty mountain]

lol

I like Lilian Weng's blog. She makes her posts almost like it's a paper, with references and equations...

[wooden sail]

so, a verified source

[hasty mountain]

I think she's a head engineer in OpenAI

[coral bloom]

when I did my thesis on openstack, theere were mostly blogs and low papers

but thats an issue with the field ig

equations dont make papers credible logic does

[wooden sail]

and what makes logic credible?

math enables you to communicate the logic and prove it correct

[coral bloom]

true, but yup, if you got some free time to burn do read it and lemme know 🙂

[wooden sail]

maybe during the weekend

[coral bloom]

imma keep racking my head to undesrtand dense layer

[coral bloom]

yaaay 🙂

[wooden sail]

a dense layer is a matrix multiplication plus an offset

[coral bloom]

yup, and to implement it in python

[wooden sail]

use numpy for that

def dense(input_vector, weights, biases, activation):
  return activation(weights.dot(input_vector) + biases)

where the input is of size N, the weights are of size M x N, the biases are of size M, and the activation is a function of your choice

[coral bloom]

mmmmm

On it

Thanks :))

[hasty mountain]

https://github.com/Martyn0324/NumpyNetwork

||Good luck with the mess I made||

[wooden sail]

wth is that landing page lol

there's no text, just random media pasted

[coral bloom]

\OOOO rhis looks like a mine

imma dig

[hasty mountain]

Yes

You'll find many texts in the references, though :)

[hasty mountain]

Hey guys, can someone help me with a specific question about diffusion models?
I just saw that Stable Diffusion applies diffusion over the latent space of an encoded image rather than the image itself. I want to reproduce that in my prototype, but I'm now thinking...would it make sense to use fully connected layers instead of convolution layers?

I was thinking...convolution layers are usually better when the pixel values have a relationship, right? However, the noising process is applied equally to every pixel, independent of its relationships with the nearest pixels.

[wooden sail]

the thing is that convolutions enforce "spatial invariance"

so they can produce the same result regardless of which part of the region they're looking at

dense layers don't have this property

[hasty mountain]

https://computersciencewiki.org/index.php/Shift_invariance_(Spatial_invariance)

So, convolutions are less likely to make mistakes when the input suffers small distortions?

[wooden sail]

not necessarily

i just mean they can create or detect things regardless of where they are

[hasty mountain]

Can you give an example?

[wooden sail]

if you make a network that detects whether an image contains a 1

it shouldn't matter where the 1 is in the image

[hasty mountain]

Oh... But would this apply in the diffusion process?

[wooden sail]

this is enforced by convolutions

[hasty mountain]

I mean...the idea is that the model would add noise to the image, so...

[wooden sail]

convolutions are spatially invariant by definition

if your diffusion model takes noise as an input and uses convolutions to produce dogs, then depending on the noise realization, you can get a dog anywhere in the image

[hasty mountain]

Hm... So, the idea would be...the model must be able to add noise to the image, regardless of which part of the image(which pixel values) the model is receiving?

[hasty mountain]

I think I can get that for a complete image, but what about its latent space?

https://www.quora.com/Do-convolutional-neural-networks-learn-to-be-spatially-invariant-at-the-last-layer-of-the-network-fully-connected-layer-Convolution-layers-produce-spatially-equivariant-output-but-what-about-the-spatial-invariance

So the conv layers are more or less a facilitator for the linear layers, at least in classification

[wooden sail]

not sure i understand your question

i just wanna say that, due to how convolutions work (take a convolution kernel, slide it along the image, multiply, add), it applies the same convolution kernel everywhere in the image

this enforces the assumption that we want to produce the same result everywhere

convolutions are spatially invariant, it doesn't matter what you apply them to

[lapis sequoia]

"Train a really simple model that can predict the likelihood of a Forest Fire, given certain parameters." I have attached the dataset. Can anyone guide me through the steps required for completing this project?

no, how long will it take me to learn them?

[serene scaffold]

@lapis sequoia your three features are oxygen, temperature, and humidity. Can you see from looking at them how they could determine if there's a fire?

[lapis sequoia]

i know basic python

[lapis sequoia]

more oxygen, higher temp, less humidity?

[serene scaffold]

This is the wrong question to ask.

[lapis sequoia]

whyy

[serene scaffold]

You don't need to memorize all of sklearn to solve this. And even if you did, there's no way to estimate how long that would take.

[lapis sequoia]

ohh, so what all should i learn before starting this project?

[serene scaffold]

You can start right away and learn stuff as you go along

First you want to use pandas to read the data

[lapis sequoia]

starting from scratch

[lapis sequoia]

that sounds good

[wooden sail]

since you're unfamiliar with all of this, i think the easiest thing to try first is linear regression

[lapis sequoia]

so what should be the order of steps i need to follow?

[lapis sequoia]

can i learn that from youtube? in a week?

[wooden sail]

idk, i would always prefer reading from a book or something like that. and learning pace is up to you

[lapis sequoia]

okayy, a statistics book?

[wooden sail]

sure, that could be

[serene scaffold]

There's no way to estimate how long any of this will take. What's important is that you keep a positive attitude about learning and find satisfaction in accomplishing intermediate steps

[lapis sequoia]

i have to submit the project by 24th of this month. I really want to do this. But I dont know where to start

[wooden sail]

there's like 3 different things you have to learn and they're not directly related to each other

take your pick

there's pandas, there's linear regression, and then how to do linear regression in python (which could be with numpy, sklearn, etc)

[lapis sequoia]

okayy, thanks a lot!!!

[arctic saffron]

Any suggested datascience/ml project after completing a course?

[drifting kelp]

Hello, why my program is crashing when calculating a matrix 16000by16000 using sparse array?

I suppose that would be a small matrix using this library.

[vocal aspen]

I have this stackoverflow issue open - https://stackoverflow.com/questions/75752140/installation-of-transformer-library-not-working-in-apple-m1?noredirect=1#comment133635681_75752140

Does anyone have any ideas on how to fix this?

[cosmic kraken]

Hello:) is there a way to make superplots with python?

[serene scaffold]

what's a superplot?

[cosmic kraken]

Where basically you combine in one graph datapoints per experiment (each replicate has a color) and the average of that experiment

You combine several experiments where you have individual datapoints per experiment and the average of those

[wooden sail]

so i think the name more commonly seen for this is "violin plot" or simply "error bars" (like in a box plot or a variation of it)

you can do these with matplotlib

https://matplotlib.org/stable/gallery/statistics/violinplot.html take a lookie here at the bottom

[gentle hound]

Do you use excel? Do you know how can I normalize this pivot table?

I don't wanna use pandas since I wanna see colors and stuff, I've been using it but at the end I thought excel was more practical

@lapis sequoia If you want we can learn together 🤔

[serene scaffold]

we usually use pandas instead of excel

if you're just using excel, it isn't necessarily a python question anymore

[gentle hound]

I know, sorry, just found a data science discord, since surprisingly excel ones didn't seem to pop up

[grand warren]

is 20 thousand total images enough for ocr model?

or is it too much

[mild dirge]

There is no such thing as too much for machine learning

More data is more good almost always

[grand warren]

oh

btw i tried your idea

it actually made it better

altho still not perfect

still couldnt detect the bottom text but theres some improvement so thanks

[mild dirge]

That's pretty cool!

[olive stone]

Hey,
I keep getting this error:

Could not locate cublasLt64_12.dll. Please make sure it is in your library path

Any idea?

[austere swift]

cublas is the blas package that's included with cuda, so that error is likely due to an improper install of cuda

[olive stone]

Is this .dll supposed to be packed with cuda_11.5.0 or cuda_12?

[fresh cave]

matplotlib is a good library for data science?

And a good library for machine learning?

Im a newbie in this

[wooden sail]

matplotlib is only a plotting library. the data science and ML has to be done with other tools

[frosty ore]

Any face detection models that won't detect cartoons? Have tried Retinaface, it detects cartoons 😦

Dlib seems guilty of this as well.

[serene scaffold]

There aren't any end-to-end libraries for data science. There are a bunch that do different things.

[austere swift]

the _12 at the end would imply that it's packed with cuda 12

[floral seal]

I'm using a CNN to generate audio, the highlighted section is generated, the beginning of the clip is sample data (a seed for the generation)
Why is it that my audio just grows in a nasty wave form like this?
Its trained on music

[scarlet kite]

Where are the best resources to learn neural networks?

[serene scaffold]

How much do you know about neural networks so far?

[scarlet kite]

basically nothing except some very basic backpropagation.

[serene scaffold]

Does that just mean that you've heard of backpropogation, or that you know what the point of it is?

[scarlet kite]

i know what the point of it is

[small stream]

cool

[verbal venture]

can someone explain what x_test and y_test are

I understand the testing datasets are the ones you use to basically validate how well your model is

but with a line of code like model.fit(x_test, y_test) what is y_test in this instance?

[wooden sail]

you would never do this. "fitting" is the process through which you update the parameters of the model

[verbal venture]

sorry, model.eval(x_test, y_test)

[wooden sail]

what you would do is predict on x_test to produce some output. this output should be close to y_test if the model was trained correctly

that is what model.eval does. predict based on x_test and compare to y_test

[verbal venture]

so y_test is the set of tensors used to validate the correctness of the model?

[wooden sail]

yep

just for completeness, whether the data is a "tensor" is a point of debate 😛 a multidimensional array being a "tensor" depends on your definition of tensor. many data science and AI people call it a tensor, but maths and physics people wouldn't

[serene scaffold]

if the term "array" hadn't already been overloaded in computer science, this wouldn't be such an issue.

[wooden sail]

it wouldn't add anything extra in this case though

i think almost any CS definition of array would be correct here

[serene scaffold]

I'm just annoyed by how "array" is used in C-style languages, because the API for those "arrays" is never designed for array arithmetic in a mathematical sense.

[violet gull]

Hi edd

[wooden sail]

there's no canonical way of doing math on an array either, to be fair

we've chosen some operations for when the array is a matrix

but it doesn't have to be a matrix

and data is not a matrix even if it is 2D

[violet gull]

u check my code 👉 👈

Matrix result = new Matrix(NUM_FEATURE_SETS, input.rows - KERNAL_SIZE + 1, input.cols - KERNAL_SIZE + 1);
        for(int i = 0; i < input.z; i++){
            Double[] featureMap = new Double[(input.rows - KERNAL_SIZE + 1) * (input.rows - KERNAL_SIZE + 1)];
            for(int j = 0; j < NUM_FEATURE_SETS; j++){
                Double[] temp = Matrix.simpleCov(input.matrix[i], kernals[j].matrix[0]);
                for(int k = 0; k < featureMap.length; k++){
                    featureMap[k] += temp[k];
                }
                for(int k = 0; k < featureMap.length; k++){
                    featureMap[k] /= temp[k];
                }
            }
            result.matrix[i] = featureMap;
        }
        return result;``` @wooden sail

[wooden sail]

a matrix performs a linear transformation, and a tensor performs a multilinear one. data does not apply transformations, so it's an array. it's acted on

[serene scaffold]

this clearly isn't Python, so please ask in an off-topic channel.

[violet gull]

its pretty close

[serene scaffold]

Nope.

[violet gull]

everything in that exists in python

[wooden sail]

it's spelled kernel btw

[violet gull]

is it actually

[wooden sail]

yep

[violet gull]

oof

i change that

[wooden sail]

but do go to OT to get your java code checked 😛

[violet gull]

is there anyone in off topic that knows what the code is suppose to do...

[hasty mountain]

[hasty mountain]

I know it's off-topic, but...
Protip: try taking a look at Eigen

[violet gull]

what?

[hasty mountain]

It's a library for linear algebra in C++

[violet gull]

i dont struggle with the math i struggle with understanding the model

all the resources online are horrible and the people that wrote them just copied it form something else

and its like putting an essay. through different google translates 50 times

[hasty mountain]

That look like just a simple convolution operation

[violet gull]

wym

[hasty mountain]

I might be tired so not getting it right, but it's pretty similar to a convolution loop.
For each row, it applies a multiplication between an element in the input matrix array and the weight in the model

[violet gull]

my code?

[hasty mountain]

Yes

[violet gull]

its not the code for a convolution in the code for a convolution layer

the code for the 2d convolution is the Matrix.simpleConv()

[hasty mountain]

Yes, and it appear that then it assigns the results to an array featureMap

[violet gull]

yes

[hasty mountain]

So it's a convolution layer

[violet gull]

yes

[hasty mountain]

That's what I said

[violet gull]

when

[hasty mountain]

Anyway, I'm a bit confused over KL-Divergence loss.
I'm trying to optimize an Encoder from an AutoEncoder:

        image = image.to(device)

        encoded = encoder(image)

        p = torch.distributions.Normal(torch.zeros_like(encoded), torch.ones_like(encoded))
        z = p.sample()

        encoder_loss = kl(encoded.log(), z)

Problem is...this results in an infinite loss, with gradients average oscillating between -10 and +10.
I've printed my encoded and my z, as well as z.log(). One example of output value for each variable is 1.7685, 0.0885 and -0.9229

Can someone help me understand what is it with this infloss? Can my model still work in this situation?

[wooden sail]

you have an input that is 0

[hasty mountain]

Ugh...damn...

Any idea of activation function to deal with that?

[wooden sail]

probably something like softmax

[hasty mountain]

Is it plausible in this case?

[wooden sail]

KLD measures the distance between probability distributions, and softmax produces something that satisfies the definition of a PDF by default

[hasty mountain]

Oh yes, thanks for reminding me.

[hasty mountain]

(I was using a Relu to avoid negative values )

[wooden sail]

idk if pythorch's KLD does the bottom part by default, the substitution of logs in the 0 case

[hasty mountain]

It doesn't.

But then, it seems the backpropagation mitigated that. It got a loss value now...a quite big one, but not infinite...but I suppose it may be better to replace that relu for a softmax anyway

Now I got something. Thanks!

[wooden sail]

nice

[sleek harbor]

How important is knowing cloud platforms for a data scientist? If I have no prior knowledge, and I'm searching to get a DS job (with no job experience), should I learn GCP or AWS, or pass on cloud platforms for now? Which is easier to learn? Is it worth it? Can anyone recommend some free resources (preferably providing some free certificate, or something I could add to my resume)?

[serene scaffold]

there are no free certificates that will be perceived as valuable by prospective employers.

That said, I use AWS at work, but AWS encompasses a lot of different services, and a team that hires data scientists would probably have a non-data scientist who specialized in cloud platforms. You would just pick up whichever services are needed for your project as you go along.

though let me emphasize that if you do not have any prior professional experience, it is almost certain that you will not be able to get a job as a data scientist without at least a bachelors.

[sleek harbor]

What if I have a bachelor's in finance, not in computer science? Does that count?

[serene scaffold]

yes, though the "data scientist" roles that you'd be considered for would probably be similar to finance analyst roles that aren't termed "data science". There's not a lot of uniformity in what responsibilities a "data scientist" has.

[silk crypt]

how do i determine the in_channel and out_channels parameter for this function? https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html

I am given the specification Conv2d [16 Filters of size = 5x5, stride = 1, padding = 0 ]

i know the kernal size should be 5

[valid heath]

is there a way to merge similar elements in a numpy array so that the values that are closer to each other are averaged out? smth like:

li = np.array([1, 2, 3, 101, 102, 103, 1001, 1002, 1003])
print(merge_similar(li))
# prints [2, 102, 1002]

i'm only wondering if there's already a function that does this, i guess to do it manually i could check the variance for each index and when there's a change bigger than a threshold store the avg of the prev elemetns in a new array

[tidal bough]

i doubt there is, no

[wooden sail]

this is similar to doing clustering, you could try a clustering alg

like k means

scipy and sklearn have implementations of this

[worldly dawn]

dbscan might be interesting there as well

[lapis sequoia]

Hello

I work with a netcdf file and data from a lidar, I have to calculate some parameters all parameters have 2 dimension (time and height) with time = 2543 and height = 324

I'm a beginner in python and xarray, I'm a bit lost, I know about the structure of xarray data. Can someone help me to complete the script.
mol_retrodiff and part_retrodiff there are nan values

1. calculation of the extinction by knowing the molecular and particulate backscatter using

mol_retrodiff = mol_data['mol_retro'].sel(time=time, height=height)
part_retrodiff = part_data['part_retro'].sel(time=time, height=height)

extinction = -np.log(mol_retrodiff + part_retrodiff)

2. calculation of the extinction coefficient knowing the retrodiffusion
   #Selection of the backscatter variables and the distance using the sel() method in the path of the ds file

range = ds.range.sel(range=slice(0, max_range))
retrodiff = ds.total_backscatter.sel(range=slice(0, max_range))
extinction = -np.gradient(np.log(retrodiff), range)

3. calculation of the optical thickness from the extinction and the retrodiffusion
   extinction = ds.['extinction']
   retrodiff = ds.['retrodiff']
   optical_depth = -np.log(extinction/retrodiff)

4. calculation of the lidar ratio knowing the backscatter and the extinction
   backscatter = []

extinction = []

lidar_ratio = backscatter / extinction

[harsh stump]

guys, quick question, before handling outliers with IQR should i drop nulls?

[keen kestrel]

The formula is written in that page, scroll down to Shape section

[silk crypt]

i think i moved past that but thanks

but can u take a look at the question i just posted in the help channel

https://discord.com/channels/267624335836053506/1086201906906550272

here

[woeful river]

hi, I have a problem when I use matplotlib
ModuleNotFoundError: No module named 'matplotlib'

[mild dirge]

Did you install matplotlib? 😛 @woeful river

pip3 install matplotlib in terminal

[woeful river]

yes

should try pip3?

[mild dirge]

No pip is fine

Are you using a venv?

Using pycharm?

[woeful river]

no, I'm using anaconda promt

already using a venv

[mild dirge]

I'm not very used to anaconda. Maybe try conda install -c conda-forge matplotlib

[mild dirge]

Oh this is important, you might have installed in in something other than your venv

You need to make sure to install it inside your venv

[woeful river]

the installation is done

but the same error

I have this output

already existed

%matplotlib inline
import matplotlib.pyplot as plt ?

the same error

[wooden sail]

maybe you can change the kernel being used

[woeful river]

the directory where jupyter I think is it

[woeful river]

I just installed matplotlib on vscode and it working:))

[mild dirge]

Cool! weird that it didn't work with anaconda

Not sure what caused it :/

[wooden sail]

anaconda always loads the base environment by default, maybe that was it?

[woeful river]

maybe because in my env folder I have another folder where I have jupyter files

[wooden sail]

you have your code files inside the env folder?

[woeful river]

no, is on Pandas folder

[misty flint]

bro i had to do that once in my DL class

bad times

[wooden sail]

how deep are we talking though

[misty flint]

this is a good resource to refresh yourself on the math https://minitorch.github.io/module1/backpropagate/

[wooden sail]

gradients for a specific example, or making the computation graph and implementing autodif?

[misty flint]

i find the diagrams helped me lol

[wooden sail]

it's an efficient way of performing math operations and their derivatives

[misty flint]

np bud

check out that link first. i think it will help solve your immediate problem. then you can move on to diving deeper.

like check out the bottom bit

[wooden sail]

indeed. the differentiation part is arbitrarily difficult depending on how good you want to make it though

an easy solution is to use sympy. this is also super slow

[grand warren]

i need some advice. so ive been learning python for a while now and i started learning tensorflow a few months ago. i have learned how to make linear regression models neural network classification models and cnns. though i feel stuck and idk what to do next...

[wooden sail]

why don't you try putting what you've learned to the test? try a couple of kaggle challenges

[grand warren]

i did try

i tried making an ocr

but its kind of uh bad

interestingly i get good accuracy with test data

but when i try an image on it i get weird results

[wooden sail]

maybe the image you tried it on needs some preprocessing?

[grand warren]

maybe idk

tho still what should i do next

i wanna go advanced

[wooden sail]

that sounds like a good thing to do next. your network isn't working yet

figure out why

[grand warren]

oh

[dull flare]

hey i recently started studying for ML / DS and for that i started with statistics

but what next

can i get a road map for ml /ds please

[glossy zenith]

Sorry that i ask such a nooby question^^ but do you think its a good idea to validate my AI Model after each Epoch like this

I also thought i should just start deleting the Model after at least Epoch 5...what do you think?

def validation(model=None):
    vali = {"NSFW": r"K:\Pre_AI\Validation\NSFW",
            "SFW": "K:\Pre_AI\Validation\SFW"}
    results = {"NSFW": 0,
               "SFW": 0}
    for Genre in vali:
        correct = []
        for image in os.listdir(vali[str(Genre)]):
            prediction = predict(vali[str(Genre)] + "\\" + str(image), model=model)
            prediction = torch.sigmoid(prediction)
            collect = []
            for i, value in enumerate(prediction[0]):
                try:
                    collect.append(float(value.item()))
                except IndexError as inx:
                    print(inx)
            collect = numpy.asarray(collect)
            biggest = collect.max()
            for i, val in enumerate(collect):
                if collect[i] == biggest:
                    if int(i) + 1 == int(label):
                        correct.append(test_Ord + sep + item)
                        break
        correct_item = (len(os.listdir(vali[str(Genre)])) / 100) * len(correct)
        results[str(Genre)] = correct_item
    if results["NSFW"] <= 90 and results["SFW"] <= 90:
        return False, False
    else:
        return str(results["NSFW"]), str(results["SFW"])

#------------------------------------------

        NSFW, SFW = validation(model=model)
        if NSFW is False and SFW is False:
            Important["breaking"] = True
            break
        torch.save(model, Net_to + sep + ModelName + ".pt")
if Important["breaking"]:
    os.remove(Net_to + sep + ModelName + ".pt")

[grand warren]

what does nsfw stand for? :-:

[glossy zenith]

NSFW means "Not safe for Work"

The Model should be used for a Content-Filter^^

[grand warren]

oh makes sense

sorry tho i dont know pytorch

[glossy zenith]

no problem^^

[daring oasis]

Hey! Does anybody have an idea of what could make these artifacts?
So I'm plotting the filled contour of a family of functions defined on the unit disk. For small values of the parameter it's fine. But when it increases, it becomes nightmarish and I get those weird discontinuities...
I'm not sure it's worth defining the family of functions in question, those are a bit technical... I'm just wondering: anybody has ever seen that kind of behaviour before and what was causing it?

[wooden sail]

i do think it depends largely on the function

things that come to mind are numerical stability. for example if your function involves a ratio whose value is well behaved, but the numerator and denominator independently get very large or very small values, you can have issues with floating point precision

but it's kinda hard to say without looking at the funcs

[daring oasis]

Ah that may actually be a reason indeed...

I manipulate this kind of functions:

def dist(z1,z2):
    a, b, c = z1
    d, e, f = z2
    aa, bb, cc = a.conjugate(), b.conjugate(), c.conjugate()
    dd, ee, ff = d.conjugate(), e.conjugate(), f.conjugate()
    n, d = (a*dd+b*ee+c*ff)*(aa*d+bb*e+cc*f), (a*aa+b*bb+c*cc)*(d*dd+e*ee+f*ff)
    return np.arccos(np.sqrt(n/d)).real```
So that actually may be a reason indeed. Is there a general method to fix that kind of issues? xD

[wooden sail]

ah wait

for the arc cos and the sqrt you also have to pick a branch

so when you take the real part, the branch may wrap around

[daring oasis]

In theory, the ratio n/d is always real positive, but it does screw up with things like 1e-10j popping as imaginary part

[wooden sail]

all right

[daring oasis]

But iirc the real part of the branch is always the same regardless of the chosen branch

[wooden sail]

then it might be better to take the real part of n/d before applying the root and arccos

[daring oasis]

Ah right!

[wooden sail]

let's start with that

there's also several products of numbers by their conjugates that could be replaced by abs()**2. other things that come to mind are writing the ratio in a different way. idk if you know anything about the behavior of the denominator and the numerator. does the denom ever get close to 0?

[daring oasis]

🤔

I just did some checking, and: supposedly, when t is either 0.1 or np.pi/2-0.1, I should get the exact same plot (cuz the functions are the same), but I still get artifacts for when I take the second choice...

I'll try to see if increasing the parameter is having an effect on what you suggested then

[wooden sail]

yep. if the function has a special name btw, there's a good chance that wikipedia has a section on numerically stable versions of it 😛 it not, it's kinda an artistic process to see which things help. e.g. compute d/n instead of n/d, or do long division or separate the ratio into a sum of ratios

[daring oasis]

Thanks a lot for the insight and for your time 🙂 I'll check my code again and hopefully I'll make it work 🙂
Also, would you happen to know how to grab the lowest-level contour line? Doing a ax.contour(X,Y,Z,[0]) doesn't work for obvious reasons (there's no level zero, there's a level 1e-15 or something).

[wooden sail]

hmm how about instead of [0], doing [np.min(whatever_your_Z_array_is)]

maybe multiplying that minimum value by some factor 1 + x with 0 <= x < 1 to taste, until it looks good

[daring oasis]

If I do [0.1] instead of [0] I get this:

But you guessed it, I'd like the bottom of that valley

[wooden sail]

yeah, let's try then with [np.min(Z)]

[daring oasis]

Woops xD

[wooden sail]

oof, so close

[daring oasis]

Yeah xD

[wooden sail]

let's get artistic, then

[daring oasis]

Sure xD

[wooden sail]

[np.min(Z)*(1.001)] lol

[daring oasis]

I think I'll just interpolate between the two level lines from before and call it a day

[wooden sail]

you can do that

alternatively, you can take the Z value and threshold everything below a certain value to 0

[daring oasis]

Ah that's a clever life hack!

[grand warren]

where do u guys learn all the maths stuff

[wooden sail]

floating point arithmetic is not doing you any favors lol best of luck with this

[daring oasis]

Wdym?

[wooden sail]

in uni, from books and papers

[grand warren]

im still in high school :(

[daring oasis]

Oh then you'll see everything in due time don't worry! 😉

[grand warren]

is it ok if i do ml without maths?

[daring oasis]

What's ml?

[grand warren]

machine learning

[daring oasis]

Oh, you'll need a ton of linear algebra for this, and differential calculus too

[hasty mountain]

From my experience, you can do that, but knowing the maths makes the process more efficient.

If you don't know the math, you'll need too much trial and error to get the idea of some things, which takes time. By knowing the math, you might be able to save some time(and headache).

[grand warren]

oh

which topics should i learn?

[hasty mountain]

Idk...
@wooden sail

I'd say linear algebra and calculus, but I can't be specific.

[grand warren]

oh

[wooden sail]

yeah linalg, multivar calc, and statistics

[hasty mountain]

Oh yes, statistics... especially gaussian distributions

Also guys...just to confirm one thing...
To measure the compression rate of my AutoEncoder, I should take into account the total number of features of both my uncompressed input and my latent space, right?

So, if my latent space has dimensions 4x4x512, and my input is an image 64x64x3, the compression rate would be 4x4x512/64x64x3 = 2/3?

[wooden sail]

sounds about right

[hasty mountain]

I'm trying to make an Encoder for a diffusion model, to apply diffusion into the latent space rather than in the complete image itself, like it's done in stable diffusion. However, if I really use a latent space of 4x4x512, it doesn't seem my encoder is that much effective in reducing the dimensions...
And it seems that using a latent space of 1024x1x1 isn't working.

[hasty mountain]

Ugh...then it seems my strategy is failing...

[wooden sail]

images are usually VERY sparse in some domain

[grand warren]

btw is there any online course or tutorials u can recommend?

about maths

[wooden sail]

then again 64 x 64 is pretty small, depending on the image there might not be much to do

[hasty mountain]

There's a channel, 2brown1blue that folks here tend to recommend

[grand warren]

its 3 blue one brown i think

[hasty mountain]

It's curious...the AutoEncoder seems to work, but...when I pass the noised latent space into the decoder, instead of it recomposing a noised image, it actually tries to recompose an image without noise.
The result is that my reverse diffusion ends up generating a noised image, with more noise at each timestep

The process I'm doing is:
Train: Image ---> Encoder ---> Latent Space ---> Decoder ---> Decoded image ; MSE(Decoded_image, Image)

For the diffusion model: Latent Space ---> Diffusion Model predict ---> Noised Latent Space ; MSE(Noised_Latent_Space, Gaussian_noise)

Sampling: Random Noise(latent space) ---> Diffusion Model predict ---> Reverse Diffusion ---> Denoised Latent Space ---> Decoder ---> Sample Image

Since it's a simple AutoEncoder(no generative configuration), I'm just using Mean Squared Error and no KL-Divergence. Seems to make sense...

[wooden sail]

i'm not very sure on the specifics of the diffusion part tbh, i can't really comment

there's a lot of sophisticated statistics there that i have yet to look into

[hasty mountain]

I think the diffusion part is ok, but the AutoEncoder is confusing me...

I've tested the same diffusion formulas(and model) using the original images, without the AutoEncoder, and everything went fine.

[wooden sail]

what does your architecture look like for the diffusion part

[hasty mountain]

class LatentLinear(nn.Module):
    def __init__(self):
        super(LatentLinear, self).__init__()

        self.embed = nn.Embedding(n_diffusion_timesteps, 1024)

        self.neuron1 = nn.Linear(2048, 512, bias=True)
        self.neuron2 = nn.Linear(512, 512, bias=True)
        self.neuron3 = nn.Linear(512, 512, bias=True)
        self.neuron4 = nn.Linear(512, 512, bias=True)
        self.neuron5 = nn.Linear(512, 512, bias=True)
        self.neuron6 = nn.Linear(512, 1024, bias=True)
        self.relu = nn.ReLU()
    
    def forward(self, input, time_step):
        vectors = self.embed(time_step)
        input = torch.cat((input, vectors), 1)

        x = self.neuron1(input)
        x = self.relu(x)
        x = self.neuron2(x)
        x = self.relu(x)
        x = self.neuron3(x)
        x = self.relu(x)
        x = self.neuron4(x)
        x = self.relu(x)
        x = self.neuron5(x)
        x = self.relu(x)
        output = self.neuron6(x)

        return output

[wooden sail]

where'S the noise part

[hasty mountain]

def forward_diffusion(x0, timesteps):

    eps = torch.randn_like(x0) # Noise

    samples = []

    k = sqrt_alphacum[timesteps]
    m = one_minus_alphacum[timesteps]

    for i in range(len(x0)):

        mean = k[i].item() * x0[i] # Image scaled
        std = m[i].item() # Noise scaled

        sample = mean + std * eps[i]

        samples.append(sample.unsqueeze(0))
    
    samples = torch.cat(samples, 0)

    return samples, eps

The samples is a batch with noised images at timestep T, the eps is the gaussian noise. The model receives a noised image at timestep T and must be able to add more noise to it trying to return a gaussian noise

        image = image.to(device)

        encoded = encoder(image)
        decoded = decoder(encoded)

        autoencoder_loss = mse(decoded, image)
        autoencoder_loss.backward()

        encoder_optimizer.step()
        decoder_optimizer.step()

        time_step = torch.randint(low=0, high=1000, size=(image.size(0),), device=device)

        xt, noise = forward_diffusion(encoded.detach(), time_step)

        epsilont = model(xt, time_step)

        loss = mse(epsilont, noise)

def reverse_diffusion(time_steps=1000, image_shape=(3, 64, 64)):

    xt = torch.randn((4, *image_shape), device=device)

    with torch.no_grad():

        for time_step in reversed(range(1, time_steps)):

            t = torch.ones(4, dtype=torch.long, device=device) * time_step
            z = torch.randn_like(xt) if time_step > 0 else torch.zeros_like(xt)

            predicted_noise = model(xt, t)

            beta_t = beta[t]
            inverse_sqrtalpha_t = inverse_sqrtalpha[t]
            one_minus_alphacum_t = one_minus_alphacum[t]

            time_step_sample = []

            for i in range(len(predicted_noise)):

                k = beta_t[i]/one_minus_alphacum_t[i]

                xt[i] = inverse_sqrtalpha_t[i] * (xt[i] - (k.item() * predicted_noise[i])) + torch.sqrt(beta_t[i]) * z[i]

                image = decoder(xt[i].unsqueeze(0))
                time_step_sample.append(image)

The variables:

n_diffusion_timesteps = 1000
scale = 1000/n_diffusion_timesteps

beta_start = scale * 1e-4
beta_end = scale * 2e-2

beta = torch.linspace(beta_start, beta_end, n_diffusion_timesteps, dtype=torch.float32, device=device)
alpha = 1 - beta
sqrt_alpha = torch.sqrt(alpha)
alpha_cum = torch.cumprod(alpha, dim=0)
sqrt_alphacum = torch.sqrt(alpha_cum)
inverse_sqrtalpha = 1/sqrt_alpha
one_minus_alphacum = torch.sqrt(1 - alpha_cum)

[wooden sail]

😩 sorry, i don't have the energy for this today i think lol

[hasty mountain]

lol

Understandable. I had to prepare mentally for like 2 months before studying diffusion models

I'll see if letting it run for more time may fix that(I only tested it for 1 epoch)

[grand warren]

thx

[velvet mountain]

i have been playing with setting up my own mlflow tracking server with ftp for artefacts, but ran into a damn load of issues while doing

after a lot of searches and inspection of their github, I realize that despite what the doc suggests, mlflow has quite a list of issues that seem to make working with ftp undoable

so I'm wondering: has anyone succeeded in making mlflow tracking server be a proxy for artefacts and models, with an ftp server for artefacts?

[torn marten]

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import pandas_datareader as web
import datetime as dt

from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.layers import Dense, Dropout, LSTM
from tensorflow.keras.models import Sequential

crypto_currency = 'BTC'
against_currency = 'USD'

start = dt.datetime(2016, 1, 1)
end = dt.datetime.now()
data = web.DataReader(f"{crypto_currency}-{against_currency}","yahoo",start,end)
print(data.head())``` can someone help me this seems like pretty fundamental code to be running but I am coming across an error and I have no idea how to debug it

Traceback (most recent call last):
  File "\main.py", line 16, in <module>
    data = web.DataReader(f"{crypto_currency}-{against_currency}","yahoo",start,end)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Python\Python311\Lib\site-packages\pandas\util\_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "Python\Python311\Lib\site-packages\pandas_datareader\data.py", line 379, in DataReader
    ).read()
      ^^^^^^
  File "Python\Python311\Lib\site-packages\pandas_datareader\base.py", line 253, in read
    df = self._read_one_data(self.url, params=self._get_params(self.symbols))      
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^      
  File "Python\Python311\Lib\site-packages\pandas_datareader\yahoo\daily.py", line 153, in _read_one_data
    data = j["context"]["dispatcher"]["stores"]["HistoricalPriceStore"]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: string indices must be integers, not 'str``` this is the error in my terminal

[nimble acorn]

greetings, would this be the channel to ask questions about data analytics?

[fiery dust]

is someone here familiar with streamlit? I'd like to know what do you think about it for dynamic data visualization, meaning data is coming in constantly and streamlit can manage updating all the charts, progress bars, tables, etc.

[sweet crypt]

Just wondering how many of you would find it useful: https://www.thedrive.ai/ ?

[thorn swift]

idk if anybody else uses geopandas but it is SLOW as hell

[thorn swift]

in tensorflow by default models are validated every epoch, i guess it would depend on how big your validation set is

[waxen epoch]

trying to get into pytorch, right now the only decent tutorial i see is 25 hours long @_@

[hasty mountain]

Pytorch has some written tutorials in their website and their GitHub

I strongly recommend you to learn a bit about classes and functions. Pytorch works around classes and functions that follows certain patterns(like __getitem__() and __len()__)

||To be honest, though, I learned those things while studying pytorch...and I'm still not sure if I really get the idea of them other than simply organizing your code||

[hasty mountain]

Hm... I think in the end it'll be one of those things that, in the paper I read, the researchers make it all work wonderfully, but when I try it, it just messes everything up

[queen cradle]

Based on the equation, it like something to do with projective space. Spherical harmonics, maybe? Anyway, one thing you can do is normalize: Pick a chart on which, say, z_2 ≠ 0, divide your equation by z_2^3, and then you're left with one equation in two variables. It ought to be easier and more numerically stable to find the zeros of that equation than the original one. (For what it's worth, I use the convention that capital letters denote variables on projective space and lowercase ones denote variables on affine charts.)

[winter drift]

Hi I'm kinda struggling to get openai gym running in my vscode

[velvet mountain]

nobody on this?

[glossy zenith]

Currently its 232 per Folder

[sick otter]

How to save OpenAi embeddings and use them in Nodejs/Reactjs/Nextjs? Hdf5 or Pickle or JSON?

[hard birch]

Does anyone know how maximum drawdown of a time series is calculated

what are the peak and troughs in reference to?

all time or a specific window?

[daring oasis]

Yep this has to do with projective spaces 🙂 But I'd rather not look at it in a chart, because RP(2) and CP(2) are compact, and R² and C² are not! I need to understand the bahaviour at infinity too. I mean, I could look at two charts, but that wouldn't allow me to visualize properly what happens 🙂
The disk is actually RP(2) here, and to a point (x,y) in the disk I map it to [x,y,np.sqrt(1-x**2-y**2)]. In fact, what I'm trying to plot is not supposed to be continuous.
I'm trying to scan all points in RP(2), and either plot them or not depending on if a property is satisfied. What I did is I took np.abs(...) and plot the contour of that, where it vanishes if and only if my property is satisfied... I know, definitely not optimal, but I couldn't think of a fast way to do it other than this!

[boreal gale]

max drawdown is basically the maximum value of the drawdown series
max drawdown itself is a scalar, drawdown series is a series of values

drawdown series is basically the series of values calculated by the cumulative maximum - current value

"peak" and "troughs" are just terms used when defining drawdown in hand-wavy way, i think you would be best served by considering the actual mathematical definition.

[misty flint]

pytorch 2.0 just released

[tidal bough]

huh

40% faster??

and it just works, on commonly available models??

sounds like magic actually

On a desktop-class GPU such as a NVIDIA 3090, we’ve measured that speedups are lower than on server-class GPUs such as A100. As of today, our default backend TorchInductor supports CPUs and NVIDIA Volta and Ampere GPUs. It does not (yet) support other GPUs, xPUs or older NVIDIA GPUs.
ah, I see, magic only available for the ruling class 😔

[olive stone]

Hey, how to normalize a huge dataset of images?

[grand warren]

use image data generator

test_datagen=IMGD(rescale=1./255)

something like this

from tensorflow.keras.preprocessing.image import ImageDataGenerator as IMGD

train_datagen=IMGD(rescale=1./255)

train_data=train_datagen.flow_from_directory(train_dir,batch_size=32,class_mode="categorical",seed=42,target_size=(img_height,img_width))

[thorn swift]

Try it with a single symbol (btc) first

[queen cradle]

For the elliptic curve you're looking at, a more conventional picture is in the affine chart z_2 ≠ 0. In that chart, the equation becomes equivalent to (z_1/z_2)^2 = (z_0/z_2)^3 - (z_0/z_2); in affine coordinates, y^2 = x^3 - x. To plot this, factor the right-hand side as x(x - 1)(x + 1). Notice that the right-hand side is non-negative precisely on [-1, 0] and [1, infty); so pick a lot of x values in those intervals, take their square roots, and there's your y. You'll get a standard picture of a two-component real elliptic curve.

Even if you don't want to plot it in the affine chart, you might still want to do some computations there. Find a bunch of (x, y)'s as above; in projective coordinates these are [x:y:1]. If you set lambda = sqrt(x^2 + y^2 + 1), then (x/lambda, y/lambda, 1/lambda) is a point on the positive unit hemisphere like you're looking for.

The chart z_1 ≠ 0 works out a little differently. It turns into -(z_0/z_1)^3 + (z_0/z_1)(z_2/z_1)^2 + (z_2/z_1) = 0, that is, -x^3 + xy^2 + y = 0. This is quadratic in y: (x)y^2 + (1)y + (-x^3). The discriminant is 1 + 4x^4, which is always positive, so the quadratic always has solutions. To plot it, therefore, you can plug in lots of values of x and apply the quadratic formula to get y. That gives you points [x:1:y], and you can scale those to get points on the unit hemisphere as before.

Those two charts should be enough. If you want you can also work out z_0 ≠ 0. You get the equation -1 + (z_2/z_0)^2 + (z_1/z_0)^2(z_2/z_0), i.e., -1 + y^2 + x^2y = 0. It's again quadratic in y, and this time also quadratic in x. So, same idea.

[arctic wedge]

Hey @open pivot!

You either uploaded a .txt file or entered a message that was too long. Please use our paste bin instead.

[open pivot]

Hi guys, I have a question about crossing petting zoo and pygame, can you help with this? I don't understand how to set up the learning environment correctly
I have a class that performs the movement and drawing of the hero, there is also a block drawing class and a maze generation class The environment should support 2 agents, and agents should also communicate with each other when it is next to (5 by 5) blocks around itself https://paste.pythondiscord.com/lemuxaxoqi https://paste.pythondiscord.com/ibayacufag

[zealous badger]

L

im getting a loss of 0.22911054325756575, after running a binary classification task using a pytorch ANN. is it decent?

{loss function is binary cross entropy loss, optimizer is SGD, trained for 200 epochs in batches of 16}

[simple tapir]

I didn't get what's wrong with this. Do you want to improve your model? (tho it's good)

[zealous badger]

im just not sure if its a decent loss to have, or if i should work towards reducing it even more

[simple tapir]

it's good like that

But if you want to reduce it, you may want to use a different optimizer such as Adam

[zealous badger]

oh , i'll try that

[hasty mountain]

All those crazy things, and I still have to resort to keras when I want a decent one-hot encode function

[hasty mountain]

||Friday: Martyn releases Inception TrimNet for Drug Discovery ||

[hasty mountain]

Just kidding, my model is horrible so far. Worse than random guessing.

[austere swift]

I now pay the early adopter tax as it doesn't support ada gpus

i do have access to an A6000 though, so it might be interesting to run some experiments to compare ampere A6000 with torch.compile vs 4090 without torch.compile

[exotic verge]

Hi everyone, has anyone worked with identifying key points on the image? Why can't the combination of the SIFT detector and the ORB descriptor work?

[serene scaffold]

I'm not an image AI person, but I think what you're looking for is "saliency detection"

[exotic verge]

Uh, not really. And by the way, I'm not working with AI, but with machine learning. But thanks for the interesting thought, maybe I'll use saliency detection in the future development of my project

[serene scaffold]

machine learning is part of AI.

[exotic verge]

I understand) it's just that many people share these concepts

[serene scaffold]

there's AI that isn't machine learning, but it's incoherent to say that you're "working with machine learning that is not AI".

[magic mantle]

im creating a regularization system that calculates a value against l1 regularization and im wondering if anyone has any experience with how modifying the l1 value directly after its already been determined by lamba effects fitting on strict datasets?

trying to conserve my rented gpu time as much as possible to all my testing is being done quite slowly on my hardware until i nail down a training method

also; if anyone has any experience with a system that opimises l1 regularization on the fly based on discrete validation checks performed through batches so that you dont have to have a phase where the model isnt training, and is just validating id be ecstatic to learn about it cause right now the script im building for that is hurting my soul

[queen cradle]

It's not clear to me how you're optimizing the constant. A validation set? Cross validation?

[magic mantle]

im also expecting theres a more efficient way to do it than having to divide off a huge section of good data in the dataset just for continuous validation checks

@queen cradle the l1 constant?

[queen cradle]

Yeah.

[magic mantle]

i have two values, one is produced by a system that calculates a weight modifier using variables including the average weight of the data used in the batch and the current value of the weight thats being passed to the model, that gets passed over to the script which is performing validation checks through batches by passing the model validation tokens mixed in with its training data, only marked to be solved, it takes the second of the values, ie. the value it generates from the models performance on groups of checks done at intervals seperated by epochs and calculates that against the dynamic weight variable and then uses that to calculate against L1 to modify it

and its all very rough and super new so i know its extremely suboptimal in terms of compute

though thankfully my data set is very easily stored and calculated and optimized, so that takes a lot of load off of it

[queen cradle]

This sounds unusually complicated. At first I thought you just had a fixed hyperparameter lambda that you were hoping to optimize. Now I'm wondering whether

the script im building for that is hurting my soul
is overcomplicated.

[magic mantle]

probably

[queen cradle]

It sounds like, based on what you said before, you're trying to be continuously training your model. I think that's the wrong thing to aim for. You should aim to continuously use your hardware.

[magic mantle]

sorry im a little fried right now because ive been doing word math in python for a few days, you saying i should be aiming to do it in smaller chunks, and spend more time validating and testing it?

[queen cradle]

It sounds like you have been developing a pipeline that tries to do two things at once. I think you should do only one thing at a time.

As long as you are using your hardware as much as possible, you will get the same performance.

It is possible that you may get better performance by doing one thing at a time because you may be able to write simpler and faster code.

[magic mantle]

my worry is that if im not updating my l1 to react to the data and validation on the fly, ill wind up spending more time training against under/overfitting

the dynamic variable is calculated beforehand and appended to the dataset itself

and the validation checking system is only running intermittently, so as not to bias the training data too much with the validation data

[queen cradle]

My belief is that you shouldn't worry about this. You are probably investing more hours of effort in creating your system than you would spend optimizing a slightly incorrect hyperparameter.

[magic mantle]

the actual values to be calculated against l1 are just variables updated by the dataset and the validation checking system so its not running a whole lot more than two mathematical expressions occasionally and one most times

well ive been hitting this wall, i think its because my dataset is too objective, or strict or something where the model is having a very difficult time avoiding under and overfitting

[queen cradle]

You may need a different model, then. Can you share more information about your data and model?

[magic mantle]

and i really dont want to submit to the belief that i need to train it on another model

[queen cradle]

If your model isn't fitting correctly, then either your data is bad, or your model is inappropriate, or both.

[magic mantle]

to be clear i mean, im fine changing my actual model

i meant using a seperate ai to help teach this one general skills so it can react to the dataset more dynamically

its most likely the actual seed of my model tbh

but sunk cost fallacy, plus having my own custom regularization methods is cool, and thats worth noting

[queen cradle]

If your results depend strongly on the seed, then there's something wrong.

[magic mantle]

operator error if not the algorythm, either that or i did something dumb syntaxwize somewhere

ive only been working on it for a few days not counting a couple training intervals so its not very clean yet

i hope its not my dataset i wrote a bunch of bullshit to sort that too lmao

honestly its probably because ive automated the process of weighing and sorting the dataset

im betting thats where the issue is coming from

i need to look through that, thanks you put me onto a problem i was solving in a bunch of wrong ways i think

[worldly dawn]

how does that gets the thetas correctly almost the same thetas of sklearn module and diffrence of JCost between them is 0 : py def getThetas(self,X, y): X = X.reshape(-1, 1) X = np.hstack((np.ones((X.shape[0], 1)), X)) theta = np.linalg.inv(X.T @ X) @ X.T @ y return theta[0], theta[1]

[young stream]

I have a machine learning model written with python which is basically analyses sentiment of texts. And I want to user can enter a link for example Amazon product comments and can see the result of sentiment analysis of comments. How can I deploy this machine learning model into web ? I don't know web stuffs 😦 Any idea ?

[serene scaffold]

I don't think Amazon wants you scraping their reviews, unfortunately.

[young stream]

Amazon was just an example but someone helped me

Flask and keras can help

someone who helped me said your model can act an api

[serene scaffold]

you should be able to just use requests for the scraping part. or possibly beautifulsoup. flask is more about getting incoming requests.

and you don't have to use keras, unless that's what you already used to make the model.

[young stream]

actually I will use youtube api to get sentiment analysis of comments

[serene scaffold]

great 👍

[young stream]

thanks anyway 🙂

[umbral charm]

https://gyazo.com/837fed25ceae90a74e3084e2ac6b6ab5

Why do i get this error

Or these errors, for context im trying to plot a graph of a planets trajectory using its orbital radius and orbital period and than using Euler method to find out next pos ect

[queen cradle]

Probably something is numerically unstable. You're getting overflows, which turn into infinities, which cause NaNs.

[umbral charm]

So im diving by 0?

[queen cradle]

No, you're probably computing infinity minus infinity.

It could be division by 0, but that wouldn't be my first guess.

Have your script print out the numbers it's working on and I bet you'll see the problem really easily.

[umbral charm]

Yea i was gonna say im not diving by 0 so idk how i got that, apparently tho if the numerator becomes so large python just makes the denom 0

is this true?

[queen cradle]

Python uses the IEEE floating point standard. This is not a Python-specific issue.

I would guess that it's numerical instability. I think I heard once that that was a problem for simulating orbits.

[umbral charm]

Would really love this, problem is my list is 40,000 integers

[queen cradle]

Can you calculate the total energy of the system at every time step? That would tell you if numerical instability is causing problems.

[umbral charm]

energy? like as in 0.5mv^2?

[queen cradle]

Yeah.

[umbral charm]

How would that tell me the instability?

[queen cradle]

Conservation of energy tells you that, in exact arithmetic, the total energy would never change. On a computer, it'll go up and down by tiny amounts because floating-point arithmetic isn't exact. Sometimes, when numerically solving a differential equation, these errors can cause feedback loops that make things blow up. Checking the total amount of energy helps you know whether that's happening.

[queen cradle]

You can also just check all the numbers for infinities and NaNs at every step. Anything where np.isfinite returns False is questionable.

[umbral charm]

Hm that is very true

So couldnt i just print out the velocity at each point instead of energy, coz orbital velocity should be constant

in theory

[queen cradle]

Sure, that works too.

[tidal bough]

yeah, R probably becomes very low, and for that to happen the system must have lost a lot of energy

[umbral charm]

Thats the thing, my code doesnt change R

R is constant

[tidal bough]

hmm

[umbral charm]

def integrate_Euler(planet, delta_t):
    Pi = np.pi
    R = Planets[str(planet)]['radius']
    T = Planets[str(planet)]['period']
    time = 1000
    steps = int(time / delta_t)
    x = np.zeros([steps])
    vx = np.zeros([steps])
    y = np.zeros([steps])
    vy = np.zeros([steps])
    x[0] = R
    y[0] = 0
    vx[0] = 0
    vy[0] = 2 * Pi * R / T
    for i in range(steps - 1):
        vx[i + 1] = vx[i] - 4. * math.pi ** 2 * x[i] * delta_t / R ** 3
        vy[i + 1] = vy[i] - 4. * math.pi ** 2 * y[i] * delta_t / R ** 3
        x[i + 1] = x[i] + vx[i] * delta_t
        y[i + 1] = y[i] + vy[i] * delta_t
    positions = np.array([x, y])
    return positions

Planets is a dictionairy and radius - oribtal radius and period is the time it takes

[tidal bough]

this equation of motion is a = -4 π^2 pos*dt/R^3. that should result in oscillations

which would look like an ellipse in the general case, sure

euler's method is very inaccurate, so maybe it blows up the energy anyway? somewhat strange, though

[umbral charm]

Yea i know

We're plotting Eulers against Eulercrom

Eulercrom gives a nice circle tho

its not even elipitical

[velvet mountain]

(aside note: you should vectorize your main logic)

[umbral charm]

English pls

[tidal bough]

that'd look a bit more compact with e.g. py pos = np.zeros((steps,2)) vel = np.zeros((steps,2))
, yeah

[velvet mountain]

it's correct english...

[umbral charm]

https://gyazo.com/1aed2831cd6830df151e47295326bf5d

this is what my Euler method looks like

This is of earth so in theory it should never exceed 1 on both axis

but its to the 85'th power

no clue how this happends

[umbral charm]

Yeas but what does it mean

[velvet mountain]

write with matrix operations on multi dimensional arrays (and get rid of the python for-loop)

[tidal bough]

there's an easy way to check if it's due to instability: halve the timestep and see if the result changes.

[queen cradle]

Instability can be really painful. It can be hard to detect. But when changing the timestep dramatically changes your results, that's a really good clue.

[umbral charm]

Currently time = 1000 and Delta_t = 0.01, ill change delta_t to 0.02 and see what happendsw

https://gyazo.com/38207bb34984e31367642f6bd2456cc4

i now get 170th power

[queen cradle]

170 = 85 * 2 and you doubled the timestep. Hmm...

[umbral charm]

HMmm

Its capped

[velvet mountain]

my point is: dont fix rigth now an unstable code that is deeply ill-written

first write it properly and see if it's still a problem after numpy will do its tricks for computing with near-degenerated matrices

then continue debugging

[umbral charm]

at 10^308 tho

Thats when i get the Overflow error

[queen cradle]

Yeah, that's where IEEE doubles run out of precision.

It's actually 2^1024.

[umbral charm]

Ahh ok this makes sense

[tidal bough]

I've tested your code and yeah, pretty sure it's precision issues - lowering the timestep makes it more of a circle instead of a spiral.

[umbral charm]

? for Euler method?

[tidal bough]

the delta_t, yes.

[umbral charm]

Yea indded it does

Takes so long to run on my computer too

[velvet mountain]

that can also be mitigated with getting rid of the loop

[queen cradle]

That's because your code isn't vectorized.

[umbral charm]

Would be vectorizing change the formula

or not, coz if its wrong i still want it to be wrong

[queen cradle]

No, it wouldn't.

[tidal bough]

how are you planning to do that?

[carmine cedar]

Quick data science question. I am working on an exercise in the 'Introduction to Statistical Learning (2e)' and want to understand the answer to a question.

For the above question, the internet states that the answer is the highlighted choice. But, I can't understand the interpretation of the interaction term. I created the table to get a better understanding but the table seems to indicate that college students who have high GPA will have on average $10k less in salary than high school students. What am I missing?

[velvet mountain]

i said it already: it can be written with a matrix operation

[umbral charm]

So i can vectorize it make my code run faster and more 'precise'

[tidal bough]

...can it? unless you mean, like, expressing this linear DE as a matrix and exponentiating it

actually, that wouldn't help since one needs the intermediate values, not the final result

[velvet mountain]

I talk about this

[tidal bough]

so am I. I don't see how this can be written without a loop over range(steps) while still being euler's method.

[umbral charm]

So erm, How would one go ahead and vectorize this?

[queen cradle]

I think it does need a loop over the steps, but each loop iteration can be a single matrix-vector multiply. It should be a lot faster.

[tidal bough]

@umbral charmcheck this out, by the way:

[queen cradle]

Instead of keeping separate x, y, vx, vy arrays, you should have one array of shape (steps, 4) (or (4, steps)). Column 0 (or row 0) would represent x, the next would represent y, and so on.

[umbral charm]

OH

so instead of like it doing 4 seperate calculation it does 1 matrix one?

[queen cradle]

Yes!

[umbral charm]

oofo

[tidal bough]

that's true, sure

[queen cradle]

It might sound silly because matrix multiplication feels like it should be a more expensive operation. But when you're writing in Python, that's not necessarily true.

[tidal bough]

tbh if I really wanted performance, i'd slap a @numba.njit on this. would make much more of a difference, since it'd actually speed up the loop itself, too.

[umbral charm]

isnt matrix multiplication just a lot of tiny multipactions

[queen cradle]

In your original loop, it computes 4. times math.pi squared times delta_t divided by R cubed every time. And all those require interpreting Python bytecode.

[umbral charm]

What does .njit do

[tidal bough]

numba compiles simple enough python functions to machine code

and stuff that just does a bunch of math on numpy arrays, like yours, counts.

[umbral charm]

OOF

I need to learn about these

ANyway thank you guys i shall implement these tommorow

BIg thanks!!

[queen cradle]

Good luck!

[tidal bough]

this is about 10x faster indeed

@njit
def integrate_Euler(R, T, time, delta_t):
    steps = int(time / delta_t)
    pos = np.zeros((steps,2))
    vel = np.zeros_like(pos)
    pos[0] = [R,0]
    vel[0] = [0, 2 * np.pi * R / T]
    omega_sq = 4.0 * np.pi**2 * delta_t / R**3
    for i in range(steps - 1):
        vel[i + 1] = vel[i] - omega_sq * pos[i]
        pos[i + 1] = pos[i] + vel[i] * delta_t
    return pos.T, vel.T

the matrix thing may or may not produce a speedup at this point

[umbral charm]

Thanks for this, will test TMR

[tidal bough]

the matrix approach actually seems 1.5x slower for me

@njit
def integrate_Euler_mat(R, T, time, delta_t):
    steps = int(time / delta_t)
    arr = np.zeros((steps, 4))
    arr[0] = [R, 0, 0, 2 * np.pi * R / T]
    omega_sq = 4.0 * np.pi**2 * delta_t / R**3
    M = np.array(
        [
            [1, 0, delta_t, 0],
            [0, 1, 0, delta_t],
            [-omega_sq, 0, 1, 0],
            [0, -omega_sq, 0, 1],
        ]
    )
    for i in range(steps - 1):
        arr[i + 1] = M @ arr[i]
    pos = arr[:, :2]
    vel = arr[:, 2:]
    return pos.T, vel.T

at least, the way I implemented it

[iron basalt]

If you add @njit(fastmath=True), what speed do you get?

[tidal bough]

That didn't speedup the normal one by a statistically significant amount, but seems to have made the matrix one about as fast as it. maybe. the std is like 10ms here on a mean of 70 here.

[iron basalt]

If you are using timeit make sure to add the startup code that calls it once.

So it's jitted.

[tidal bough]

i did do it

otherwise timeit would have complained to me about outliers on first run 🙂

[iron basalt]

Can arr=np.zeros become arr=np.empty here?

If so I wonder what difference it makes.

Probably not much since memory comes zeroed.

[umbral charm]

What if I was to just run it off my GPU

Is there a way to do that

[tidal bough]

i doubt it'd help, since it's fundamentally an iterative algorithm

GPU are good at parallel stuff, they are kind of like a cpu with hundreds of not-very-impressive cores.

[tidal bough]

empty is in fact a bit faster I believe, but in this case I doubt it matters at all - the time is mostly spent looping

[iron basalt]

Yeah, it depends on how it fills with zeroes. I'm assuming they are doing it the fast way. The slow way would be to rely on a for loop that sets it all to zeros, then there would be basically 2N iterations.

[umbral charm]

I can use this on Eulcrom too?

I’ll just have to change some stuff

[tidal bough]

yeah, for the euler-cromer it's maybe easier to write it in matrix form
nevermind, the obvious way is shorter

[umbral charm]

Bro thanks

No clue how you guys know all this

[tidal bough]

thanks to helping you I looked into symplectic integrators and ended up learning about this very cool one

which is indeed really powerful:

error in energy consistently at the order of 10^-15, nice

[violet gull]

Edd

where Edd

[serene scaffold]

what would you ask Edd if he were here

[violet gull]

about my strange training graphs

[serene scaffold]

last time I talked to Edd, I asked him if he likes the diabeetus cat

[violet gull]

my Edd 💔 😭

@serene scaffold wait do u know AI

[serene scaffold]

yeah

[violet gull]

what this

[serene scaffold]

that looks pretty fucky

[violet gull]

yes very fucky wucky

[serene scaffold]

why don't you just stop before it spikes way up

[violet gull]

why does it spike up

[serene scaffold]

idk, do you have a dynamic learning rate?

[violet gull]

no

its static

[serene scaffold]

I've never seen that happen with a static learning rate.

[violet gull]

well it is static

this is when i ran it again

[serene scaffold]

what does the model do

and what kind of model. and what hyperparams

[violet gull]

image classification

[serene scaffold]

is it a neural network with convolutional layers, or what

[violet gull]

yes

[serene scaffold]

hmm.

[violet gull]

        model.layers.add(new ReLU());
        model.layers.add(new MaxPool(3));
        model.layers.add(new ConvolutionLayer(15, 1, 3));
        model.layers.add(new ReLU());
        model.layers.add(new MaxPool(3));
        model.layers.add(new Flatten());
        model.layers.add(new DenseLayer(32));
        model.layers.add(new ReLU());
        model.layers.add(new DenseLayer(2));
        model.layers.add(new Softmax());```

this is the model if it helps

image size is 26x26

120 images

4 batches

[serene scaffold]

that's not python

[violet gull]

would you like me to remove the "new" so you are able to read it

[serene scaffold]

what are you even using to make it?

[violet gull]

why does that matter

[serene scaffold]

I'm just curious

[violet gull]

java

[serene scaffold]

but like what java library

[violet gull]

none

am big boy i dont need no libraries

[serene scaffold]

uh okay

[violet gull]

so why the training grap do the unga bunga dance

[serene scaffold]

you still haven't ultimately shared enough information for us to figure that out. and now that you've revealed that you're not using a widely-used NN library, we also have to rule out the possibility that your implementation of those classes is incorrect.

I don't want to have to sift through your implementations for those layer classes, so I'm getting off the train here, unfortunately.

[violet gull]

ok

although im pretty sure they are all correct

it was able to minimize the loss

[plucky bolt]

Assuming this is the place where all the Data Science/Machine Learning folks hang out, I need some help. I want to learn the subject and practice it's application with Python. I don't know Python yet and need to learn Data Science too. Is there a decent get started thing you guys can refer me to? Thanks 🙂

[violet gull]

maybe learn python before u attempt data science

[plucky bolt]

Speaking of that, do you know any resources that focus on the kind of application that'd be suitable for the data science stuff? I am saying that because from my experience with other languages, there's endless usage cases of most languages that most programmers don't even get involved with. People tend to gravitate towards certain applications.

[violet gull]

tensorflow or pytorch

numpy

[plucky bolt]

I think I've come across that while trying to plot stuff in C++.

[burnt falcon]

I have a combinatorial optimization problem, and I'm looking for friends. Has anyone ever heard of ortools, pyomo, or pulp? Seems like few if any python people use them

[velvet mountain]

assuming you already master at least 3 langs and among them, 1 dynamic and 1 oop,

I'd suggest to not loose time learning python as such

start a Jupyterlab notebook, try to read a csv with pandans, display basic statistics and plots (matplotlib maybe), try a classification task (with sklearn lets say), try to store the final model with mlflow, reload it

I think u have enough keywords to get started

good practices you know from other langs apply: environments (pyenv), dependency managment (pip), unit test (pytest), typing (pyrigth), documentation (sphinx)

you'll learn enough python passively while doing that

[velvet mountain]

forgot to reply explicitly ^

[plucky bolt]

Thanks. That’s a lot of helpful pointers.

I know a few languages but not a master at any. I wish I was though!

I hope familiarity with other languages and programming paradigms will at least help me cover the basic stuff fast enough to be doing actual exercises with the language soon.

[hardy depot]

guys i need some help

so like if i increaase the size of kernals in a convolution, will the input image sizedecrease

[velvet mountain]

yeah, just try to have a specific task

here in what I suggest, it's more about scripting a bit and getting in touch with a tiny but relevant part of the ecosystem

you'll likely spend more time setting up things, than getting stuck with the language

and if ever you want to take a step back while doing, you can still dig deep into one aspect

also dont hesitate to setup pycharm ide

[hardy depot]

it will right

[north adder]

Hello everyone, im a double major in Math and Computer Science. Im going to graduate next year and i decided i want to go into data science. I will take a data science course in Fall and probably ML in spring but i want to start learning python from scratch( they didnt teach it at my uni only java and C++). Any recommendation on how to start learning ( YT, book, course online.). i also joined a data science server but i figured out before i jump into it i need to know the Baisics of python. Thanks in advance

[velvet mountain]

if you already master java and cpp and your goal is ds, I'd suggest the same as above

[velvet mountain]

here

you'll learn enough python for a start

you'll likely get puzzled by some differences with java/c++, but that will depend on you

[north adder]

im more of a theoretical person than programming and i learned that data science requires analysis skills more than it requires programming (could be wrong idk). So i know Java and C++ as basics not mastered thats why i want to focus on python.

but im a fast learner dont think i will find trouble learning python basic i already know some things from YT

Someone suggested to read "Automate boring stuff with python book"

book link : https://automatetheboringstuff.com/

[velvet mountain]

me too, but having a context has many benefits to bound the learning

[north adder]

so what do you suggest i do?

[velvet mountain]

the task I described runs through a lot of common tasks in a ml life cycle: descriptive analysis, training a model, serving the trained model as an api

it covers a lot of common tools and libs,

and is ligth enough to not need a deep understanding of python lang nor maths

then depending on you, you can dig some pars deeper

[harsh stump]

Guys im trying to handle null values of a column that's essentially label encoded, values from 1 to 6
i have approximately 240 out of 5000 rows that has nulls.

how can i fill the null values in this case? the model i will be working on eventually is random forest

[fair trail]

why was i pinged

[velvet mountain]

you can maybe use numpy.fromiter or smth like that for the missing piece,

but yeah jit migth still be nicer

recently we tried to vectorize a code in java, and we were unable to perform better than the JIT on scalar double primitive, so that kind of beasts migth be interesting takes indeed

[velvet mountain]

vectorization is not just about efficiency

in floating point arithmetic, storing a float costs you precision

using atomic cpu operations to perform as many things as possible, "all in one", can help reducing that impact

the semantic of the computation is also cleaner

the shape of the matrix could also be exploited more deeply in some cases, to phrase the same algo a bit differently and reduce the unstability.

long story short: a cookbook solution migth not always be the first thing to consider in front of a problem

[patent lynx]

Isnt the point of vectorization is just to make computers understand what you are inputting?

[wooden sail]

well, sampling alone is enough to let you achieve that. vectorization does 2 things on top of that. one is to use tools from linear algebra, which saves you explicitly writing loops. the other is that operations between matrices and vectors are parallelizable in the sense that the same operations are applied on different inputs, which is easy to parallelize both at thread level and via special instructions to the processor (e.g. AVX, CUDA)

so it's actually a lot faster, since people have taken the time to write super fancy linear algebra libraries that exploit the architecture of the processor

[patent lynx]

Hmm

Am i confusing this from pca and standardization?

[wooden sail]

it has nothing to do with those things

[patent lynx]

This looks like an advanced topic where should i look it up?

Maybe i can use those libraries to improve super long training and fitting times.

[wooden sail]

https://en.wikipedia.org/wiki/Array_programming
https://en.wikipedia.org/wiki/Single_instruction,_multiple_data

[tidal bough]

Unless you are personally writing some of the number-crunching involved, no.

[wooden sail]

if you're using numpy or pandas correctly, i.e. without explicit python loops, you're already doing it

or tf, pytorch, etc

[patent lynx]

No probs with pandas since boolean indexing makes things easier

Though pytorch and tf cross validation are very difficult

[harsh stump]

Is this classification model an overfit?

[serene scaffold]

not necessarily. you only have two classes. it might be that the properties of those classes make it easy for the model to distinguish them.

do you have access to a similar dataset?

[harsh stump]

Unfortunately not, The this is that the 0/1 ratio is 5:1 and i guess that’s why the percent is high?