data-science-and-ml

[wooden sail]

it will also depend on the hyperparameters you're using. in lasso, one on purpose gives up MSE accuracy to enforce a laplacian prior (if seen from the bayesian perspective) or enforce sparsity (if thinking about strong equivalence with the l_0 pseudo norm)

if seen from a statistical perspective, each cost function makes a different assumption on the distribution of the parameters. this, of course, influences the final result

[brisk nest]

Thanke you very much, I'll look those things up for sure.

I really appreciate it.

[green wasp]

So about the project I had in mind. Should I learn data science first, maybe using data science from scratch which I heard is good at showing how the algorithms are implemented and how they work and why, instead of trying to piece information that I don’t know together?

[wooden sail]

if you wanna know how stuff works, start with linear algebra, statistics, multivar calculus, and optimization

books on data science will cover those topics to varying depths

[brisk nest]

What books do you recommend? Preferably those that explain the things you said about laplacian prior and sparsity

[green wasp]

That’s too in depth for now

I want to know enough so I know what to use and why, but also not spend ages studying it

So that seems like a good compromise! Bjt I’ve never read that book so idk if it’s good

[wooden sail]

hmm honestly i read the stuff from papers for the most part, but this resource looks simple enough to cover the overarching idea https://bjlkeng.github.io/posts/probabilistic-interpretation-of-regularization/. the fourth reference in there seems promising, it's a book. aside from that, you could look up the original compressed sensing paper or anything else that discusses the (strong) equivalence of l0 and l1 regularization

and in general, maximum likelihood estimation

[green wasp]

Or would you recommend another source? Maybe another book or YouTube course idk.

[wooden sail]

i really wouldn't know what to recommend if you don't wanna look at the maths

[green wasp]

Nono like

I don’t mind looking at the maths. But it’s like, a fine line between good introduction and too much depth

[wooden sail]

i would interject that that is still only an introduction haha

idk, maybe try a coursera course or something. you can apply for financial support and take them for free. most of them don't cover much math at all and are only introductory

i might be biased though. if i take a step back, maybe what i'm calling introductory here is actually what most people actually do and use in industry

with which i mean that there is a lot of stuff out there in papers that is very cool and nice and in depth, and will also never be used 😛

[green wasp]

Mmmhhh

[green wasp]

Oh yeah I totally missed this bit. Do you have books that you could recommend?

[wooden sail]

maybe that 4th reference in the link i shared just above? i really wouldn't know though, since i went the other route

[shut ivy]

Hellow, I'm trying to implement a filter for my image and video processing class and I either get huuuuuuuge numbers or veryyyyyy small numbers. Any idea why?

def H(F_image, alpha, beta):
  column_values = np.arange(0, F_image.shape[0])
  number_of_columns = F_image.shape[1]

  number_of_rows = F_image.shape[0]

  u_vector = np.arange(F_image.shape[0])
  v_vector = np.arange(F_image.shape[1])

  #u_vector = np.arange(-round(F_image.shape[0]/2)+1, math.floor(F_image.shape[0]/2)+1)
  #v_vector = np.arange(-round(F_image.shape[1]/2)+1, math.floor(F_image.shape[1]/2)+1)

  u,v = np.meshgrid(u_vector,v_vector)
  u = u.T
  v = v.T
  u = np.dstack((u, u, u))
  v = np.dstack((v, v, v))
  print(np.imag(F_image))
  return np.sinc(alpha * u + beta * v) * np.exp((-1) * np.imag(F_image) * math.pi * (alpha * u + beta * v))

  [ 5.85476888e+03  3.55296309e+03 -4.77836041e+03]
  [ 5.40344149e+03 -2.44353024e+03 -3.46656033e+03]
  ...
  [ 3.00343628e+02 -2.25009298e+03 -1.66907145e+03]
  [ 3.46656033e+03  2.44353024e+03 -5.40344149e+03]
  [ 4.77836041e+03 -3.55296309e+03 -5.85476888e+03]]

 [[ 6.10761859e+03 -5.18469541e+03  1.93602889e+03]
  [ 1.96075090e+00 -6.65368034e+03  1.09187361e+03]
  [-1.51856689e+03  2.10356980e+03 -6.24885917e+02]
  ...
  [ 1.15030145e+02  2.05823374e+03  1.46883180e+02]
  [-3.44028404e+03  9.24680885e+02  5.28496389e+02]
  [-1.77653858e+03 -1.63964361e+03  2.25144694e+03]]```

Here's the formula I am using.

[wooden sail]

why is there an np.imag(F_image) in the exponential?

[shut ivy]

because j in the formula is the imaginary part of the complex number @wooden sail

[wooden sail]

j is the imaginary unit, that is just 1j in python. you put in the entirety of your image, not just j

you presumably wanna do either a hadamard product or a convolution with H, not put the image in the argument of the exponential

[green wasp]

In your experience what does a data engineer do? Is like. Data scientist a “step up” from data engineer?

Like how system engineer is a step up from sysadmin

[flat hollow]

Hi all, I have a matrix of 3D images and for each image I have calculated the velocity map in 3 orthogonal directions at each voxel. I could now sum them in quadrature and have a scalar output, but I was wondering if there are any ways/packages that can visualise 3D vectors like in my case. Any ideas?

[shut ivy]

Hi thanks for your reply. I don't quite catch what you mean by 1j.
For this filter the teacher explicitly mentioned to apply the mask "pointwise" (so no convolution I gather)

@wooden sail Ohhh nvm I get what you meant! Thank you for saving me hours (arguably days) hahah I had been fighting with this for dayssss

[brisk nest]

Hi guys it's me again. So I have a dataset of historical stock prices of a certain company that includes the whole year of 2020 with a split of 80% so meaning the training data is from January to September of 2020 and the test data being the remainder of the year. I fitted the models with the training data and ran and tested the trained models with the testing data which produced the results in the first image. But then when I added a validation data set consisting of the whole month of February 2021 and ran it through the trained model It produced the results in the second image. I was just wondering why the overall best performing models in the testing data is LASSO and it changed to Ridge regression on the validation data?

[wild pagoda]

hey everyone, how do i read base64 csv file to pandas?
ex:data:text/csv;base64,aWQsbG9uXzEsbGF0XzEsbG9uXzIsbGF0XzIsbG9uXzMsbGF0XzMsbG9uXzQsbGF0XzQNCg==

[tidal bough]

separate out the actual base64 - only the stuff after base64,, it looks like. Unencode it via base64.b64decode. Pass to pandas's read_csv (you might need to use an io.StringIO to trick it into thinking it's a file)

[wooden sail]

a model's ability to fit the training data and to extrapolate are not directly related. if anything though, using again the no free lunch theorem, it is possible to get an intuition that, the better you perform on the training data, the worse you will perform on the validation (previously unseen) data if it is very different from the training data (cannot be somehow formulated as a function of the previously seen data). that'd be what is known as overfitting

[brisk nest]

Thank you very much as always.

[celest vine]

Is anyone here

[serene scaffold]

perhaps, but you have to introduce a topic or ask a question

[odd meteor]

I know you can visualize pixels and voxels using imageio package.

import imageio

Also if you're interested in summarising the distribution of intensity values in your image, there's a function in scipy called ndi

Check Scipy.ndimage online to see if it's what you're looking for.

[lilac dagger]

Anyone got good courses on neural networks?

[charred stump]

Hi all, I'm trying to build a chatbot and I'm having trouble with finding data for my chatbot. I went through YouTube tutorials but Idk how to generate intents for question/answer datasets, I tried creating my own dataset with 70 intents and question/answers for each intent but I'm not sure if this is the optimal way of doing things. Can anyone help me with my problem?

[misty flint]

@serene scaffold broooo they want me to dockerize my model

did not have a good first time experience with docker the first time around

but maybe it will be different this time

[wooden sail]

i must interject, that emoji is a thing of beauty

[misty flint]

the kekHands?

i cant take credit

[wooden sail]

indeed

[misty flint]

its from tina huang's server

she is hilarious tho

highly recommend if you havent heard of her

former DS at faang company

current content creator lol

[wooden sail]

i've seen her be recommended to me on youtube. might check out

[misty flint]

i love her goldman sachs internship video

you really get an insight into how those types of companies treat finance teams vs. tech teams

[rose quarry]

How can I get the return value of predict() within sklearn to write to a file?

atm Im using this python with open("predictions.csv", "w") as f: f.write("predictions\n") for j in test_data: f.write(clf.predict([j])+"\n")

But that gives me this error numpy.core._exceptions._UFuncNoLoopError: ufunc 'add' did not contain a loop with signature matching types (dtype('float64'), dtype('<U1')) -> None

[wooden sail]

what data type is the prediction?

[rose quarry]

Just an integer

[wooden sail]

you probably need to cast to str before adding "\n"

or use f string

namely, ( str(clf.predict([j])) + "\n" ) or the f string equivalent

[rose quarry]

worked perfectly, thank you 🙂

How can I get to a simple integer?

since that produced results like [0.] but I want it to just be 0

wait Im dumb

[wooden sail]

if you're sure you can do so, do int(np.round(number)) before doing str

[serene scaffold]

it will be okay

the dockerfile "language" is very simple. you just need to write a command to copy the model file into the image, and whatever else you need to expose it.

[misty flint]

okay okay

thank you for calming me down stel

it really is simple syntax

i just like to escalate things when im tasked with things and feel responsible

[serene scaffold]

r/antiwork

[misty flint]

[odd meteor]

If you have the time, I'll recommend you read up on "Understanding Intents and Entities in NLU" online.

Intent

Intent is just a broad description of what a person is trying to say. Example, hi, hey, hello, waddup
Once your bot sees this, it automatically picks that this user is trying to greet it (the bot).
There are several ways someone might express this intent, but for better understanding you could code those four.

Entities

When it comes to Entity, this is second part of NLU problem ; which is to extract entities from text.
Example:
Assume someone typed "I wanna take my boyfriend out for some suya on June 10 in Lagos"

Your bot's ability to identifying suya as a cuisine type, June as date, Lagos as a location is what determines its efficiency.

You could use regex to identify intents, do intent classification, perform entity extraction.

You could use NER or Word Vectors as another alternative with ML if you find regex in this case tiring as I do 👀😭

It's just a lot to explain and type and I'm feeling lazy. But just try to check online. Resarch about:

1. Understanding Intent and Entity in NLU
2. Building Chatbots with ML

[vagrant monolith]

Hi

How can i connect one layer to another in backward direction in an RNN

?

[serene scaffold]

with pytorch or with tensorflow? (or keras?)

[vagrant monolith]

So, here is the final architecture:

I --> A --> B --> C --> O
^ |
|___________ _ ____________________________________ _ _ _ _ _ _ |

[vagrant monolith]

keras

[serene scaffold]

I don't know, but I think you've given enough information for someone else to answer

[charred stump]

Thank you so much for helping me, I'm currently using spacy for ner. However I have no clue how to automate generating(?)/identifying intents for large datasets. Can you give an example on how to use regex to identify intents in the dataset? Or if you can point me to a link that would be very nice. Once again, thank you for taking your time to help me.

[vagrant monolith]

Do you know how to with tensorflow ?

[serene scaffold]

not off the top of my head, but I can't dedicate much time to questions today, unfortunately

[vagrant monolith]

Ahh it's ok thanks anyways

[serene scaffold]

>>> a
tensor([0.0009, 0.0009, 0.0009, 0.0009, 0.0008])
>>> a * 1000
tensor([0.8850, 0.8545, 0.8545, 0.8545, 0.7935])
>>> b
tensor([-0.0030, -0.0030, -0.0029, -0.0029, -0.0029])
>>> b * 100
tensor([-0.2991, -0.2960, -0.2869, -0.2930, -0.2899])

I have a use-case where I need to rescale tensors such that the non-zero digits start in the tenths place, even if they start in a different position for each tensor. Does anyone know how to accomplish this?

It should be the same place within the same tensor.

[tidal bough]

Which element's nonzero digits? Is it okay to only look at, say, first element?

[serene scaffold]

the first non-zero digit should be the same for every element of a given tensor.

[tidal bough]

so it should be okay, then.
I'd do something like

!e

import numpy as np


def first_place(x):
    """
    Returns the position of the first significant digit of the absolute value of the number.
    Examples:
    5 -> 0
    0.2 -> -1
    0.0065 -> -3
    1e-11 -> -11
    5064 -> 3
    -0.0065 -> -3
    """
    return np.floor(np.log10(np.abs(x))).astype(int)


def rescale_sigdigs(arr):
    return arr * 10**(-1-first_place(arr[0]))

# test
assert np.array_equal(first_place([5, 0.2, 0.0065, 1e-11, 5064]), np.array([0,  -1,  -3, -11,   3]))
# use like:
print(rescale_sigdigs(np.array([-0.0030, -0.0030, -0.0029, -0.0029, -0.0029])))

[arctic wedge]

@tidal bough :white_check_mark: Your eval job has completed with return code 0.

[-0.3  -0.3  -0.29 -0.29 -0.29]

[serene scaffold]

interesting. thanks 😄

@tidal bough this seems to work, given that I'm using pytorch

def rescale_sigdigs(arr):
    return arr * 10 ** (-1 - np.floor(np.log10(np.abs(arr[0]))).to(int))

ipd.Audio(rescale_sigdigs(a_wav.reshape(-1)), rate=n_sr)

I wish the array-like API was consistent. it's annoying having to keep track of astype vs to

[tidal bough]

yeah, that's pretty weird

cupy does it right at least

[wild pagoda]

hey everyone, how do i go from the top-bottom not left-right from pandas?

10.0   40.0    5.0    7.0    9.0   20.0   70.0    6.0    8.0
14.0    4.0    6.0    8.0   20.0    3.0    5.0    7.0    9.0

[serene scaffold]

why don't we all just switch to cupy for everything

[serene scaffold]

not sure I follow. what are you trying to do?

[wooden sail]

read along columns, i think they mean. pick a column and read out its rows

[wild pagoda]

i mean, currently my code

for row in df:
  print(row)

it gonna print

id
lat_1
lat_2
lat_3

what i want is:

10.0   40.0    5.0    7.0    9.0   20.0   70.0    6.0    8.0
14.0    4.0    6.0    8.0   20.0    3.0    5.0    7.0    9.0

[serene scaffold]

you usually shouldn't be iterating over dataframes, but there is the iterrows method.

keep in mind that it will give you a tuple

!docs pandas.DataFrame.iterrows

[arctic wedge]

pandas.DataFrame.iterrows

DataFrame.iterrows()```
Iterate over DataFrame rows as (index, Series) pairs.

[serene scaffold]

@wild pagoda got it?

[wild pagoda]

not too sure

still kinda blurry for me

[serene scaffold]

do you know how .items() works to iterate over a dict?

[wild pagoda]

tbh no,

[serene scaffold]

try for idx, row in df.iterrows(): and see what happens

[wild pagoda]

oh

[wild pagoda]

thanks for the info!

i manage to make it work

[spare briar]

check out this new course by Yann Lecun https://www.youtube.com/playlist?list=PLLHTzKZzVU9e6xUfG10TkTWApKSZCzuBI

[urban prism]

Andrew Ng's courses are pretty good

[odd meteor]

You could do something like this.

import re

print('Using Regex for Intent Classification')

keywords = {
'goodbye' : ['bye', 'farewell'], 
'greet' : ['hello', 'hi', 'hey', 'waddup'], 
'thankyou' : ['thanks', 'thnx'] 
} 

patterns = {}

for intent, keys in keywords.items():
    patterns[intent] = re.compile('|'.join(keys))

print(patterns)

Print('===== Part II =====')

"""
Now that we've created patterns dictionary, let's now define a function to help us find the intent of a message. 
"""
def match_intent(messsge):
    matched_intent = None
    for intent, pattern in patterns.items():
        if pattern.search(message):
            matched_intent = intent
    return matched_intent

Part III. Response Function

def respond(message):
    intent = match_intent(message)
    key = "default"
    if intent in responses:
        key = intent
    return responses[key] 

So when someone now interact with your bot, the flow of the convo will look like

User: hello!
Bot: Hello you! :)
User: bye byeeee
Bot: goodbye for now

So after dealing with the Intent, you'd have to code Entity part as well if you're sticking with Regex. I don't like using it because it can get too long...
If you're like me that avoids regex, you could use ML to do all these too

[charred stump]

Thanks for helping me, I'll look into this.

[shut phoenix]

What's:

for f_names in categorical_columns:
  vocab = df_train[f_names].unique() 
  feature_columns.append(tf.feature_column.categorical_column_with_vocabulary_list(f_names, vocab)) 

Used for?

Getting the values then converting them into numbers?

Like a = 0, b = 1?

[thin palm]

what's up Python gang how do I use a function from another directory within my current file in Python???

a lot of unclear answers on the web

for example

    -> folder
        ->file1.py
   application
      -> file2.py (we wanna use the above function in file 1.py) to use here)```

[wooden sail]

i usually go with sys path append

[undone wedge]

You can import python functions from other directories using dot notation

import ../folder/file1.py

[thin palm]

I am doing this exact same thing but keep getting errors? What am I doing wrong? Here's my code:
import ../wbanalysis/wbanalysis/params.py

[wooden sail]

try with import sys, and then sys.path.append(...) instead?

[thin palm]

also received an error trying this

sys.path.insert(1, '/Users/munjismac/code/munjik/wbanalysis/wbanalysis')

wbanalysis being the folder

[wooden sail]

try append instead of insert and remove the first param. give the full path, too, starting at C or whatever drive it's in

also idk if python on windows allows / ? try \ instead (use f string or double backslash)

[thin palm]

python on mac

[undone wedge]

Add an init.py file to the directory and try importing without the file extension

[wooden sail]

aight, for whatever reason seeing Users made me think of windows, my bad

[thin palm]

to the directory of the second application?

[undone wedge]

to the directory containing the function you are trying to import

[thin palm]

gotcha gotcha, thanks man!

[scenic hawk]

How do i send a GET request with a URI parameter.

?

[undone wedge]

params = {'key1': '[URI]'}
response = requests.get('[URL]', params=params)

[runic zodiac]

th x and Y values are not visible can someone help

[somber prism]

guys i loaded a dataset using tf.data.Dataset.from tensor slices and my map function is like this ```
def preprocessing(x, img_path):
print(dir(x))
name1 = str(x[0].numpy())
name2 = str(x[3].numpy())
num1 = str(x[1].numpy())
num2 = str(x[2].numpy())
target = float(x[4])

    img_name1 = f'{name1}_{add_zeros(num1)}.jpg'
    img_name2 = f'{name2}_{add_zeros(num2)}.jpg'
    
    img1 = plt.imread(os.path.join(img_path, name1, img_name1))
    img2 = plt.imread(os.path.join(img_path, name2, img_name2))
    
    return tf.convert_to_tensor([img1, img2, target])

[tidal bough]

plt.style.use("dark_background") might help

that said, I think I saw a person with a problem like that and in their case their defaults were corrupted somehow

[haughty topaz]

for item in set(df_label_vector["label"].values):
    df_label_vector.loc[df_label_vector["label"] == item].drop(labels=range(1,2500), axis=1)

df_label_vector

I wish to drop the first 2500 rows for every label

I'm kinda struggling

Right now I'm getting keyerrors: [1, 2, ... 2500] not found in axis

[lavish swift]

how many of each label are there? I'm curious where 2500 is coming from?

[haughty topaz]

There's about 5k of each label

that's why I chose 2500

I'm just throwing this into a sklearn DecisionTreeClassifier and it takes wayyyyyy to long

that's why I want to remove some data

[lavish swift]

ah...my first thought was to just drop_duplicates() but that'll remove too many

[haughty topaz]

let me check

[lavish swift]

well, with drop_duplicates() as far as I know, the most you can keep is the first or last item, so you'll get ONE item for each label

[haughty topaz]

that's def not enough lol

[lavish swift]

maybe sample()

using the frac to get .5 of each?

I don't currently have a dataset to play with to know if that'll work. Sorry.

[haughty topaz]

The sample thing is actually really cool

[lavish swift]

yeah, my only concern would be that it technically COULD remove all of one label. Given the size of your data, it's not super likely, but it's going to sample the entire set and take half

so I'm wondering if there is a way to "bin" each label first, to help make sure you're sampling each label equally

[haughty topaz]

Well I could make a new dataframe and put .5 frac sample of each label in it

[lavish swift]

yep. There are definitely ways to do it. If you don't have that many labels, might be worth doing it that way. I just can't help but think there is a "pandas way" of doing it 🙂

[haughty topaz]

Still takes soooo long

DecisionTreeClassifier

[lavish swift]

I don't know much about those. As a simple test to see if something else is wrong, you could either adjust the fraction so your dataset is even smaller, or use the drop_duplicates() and make it REALLY small and see if it's still slow. If it is, maybe look elsewhere for a speed-up

[steady basalt]

first time seeing darkmode on notebook

Anyone know how to make an env other than base default active when opening computer?

I no longer use base cause I screwed it up

Cba to activate every day

[serene scaffold]

if you just want to get roughly half of your data by label, try df_label_vector.groupby('label').head(2500), and that will give you the first 2500 rows for each label.

one should rarely be looping over a dataframe, especially if you're trying to modify it.

[steady basalt]

I have never seen anyone attempted to loop through a data frame like that

[steady basalt]

Why not just use iloc

[serene scaffold]

how would they use iloc to accomplish this, considering that they want to do an operation with respect to the value in a given column?

[steady basalt]

Ah

It’s grouped

Isn’t there a conditional command you can use

For each group first 2500 index

[serene scaffold]

that's what I did.

[steady basalt]

But not dropped out of the df

I think they wanted it removed Ud have to set it equal to a new variable

[serene scaffold]

it's the inverse, because they said that their actual goal is to just reduce the size of their data.

[steady basalt]

You do nlp?

[serene scaffold]

df_label_vector.groupby('label').head(2500) means "group the df by the label value and put the first 2500 rows of each group into a new df"

[steady basalt]

I’m not sure if I shud choose nlp as my assignment

Project

It’s an option

[serene scaffold]

were you the one who was having to figure out if a given question was good or not?

[steady basalt]

I don’t think so

Nlp looks like effort compared to comp vision and rnn

[serene scaffold]

by rnn, you mean recurrent neural networks? because those are used extensively in NLP

[steady basalt]

Nope, tile series

Time

Lstm models

Well they didn’t exaclty tell us yet

But I suspect it

[serene scaffold]

LSTMs are used in NLP as well

[burnt pilot]

import pandas as pd

df = pd.read_csv('ex6.csv')
print('df = \n', df)
print("------------------------------------------------------")

#a

def func(x):
    for i in x:
        if i > 0:
            print('greater')
        elif i < -1:
            print('smaller')
        else:
            print('medium')
df['kol3'] = df["two"].apply(func)     

im trying to create a new column called 'kol3'
that for values larger than two in column"two" will print greater
for values smaller than -1 will print smaller
and for every other values 'medium'

[steady basalt]

Yea, but we’d have to choose

I’m thinking a cnn project

[burnt pilot]

can someone help me please

[steady basalt]

But that also has its own issues when you need to segment

[burnt pilot]

the info just prints out on terminal but the new kol3 is still empty

[steady basalt]

Another df loop

[serene scaffold]

pandas uses a special data model, where "atomic" operations are actually applied to the whole column.

keep in mind that printing is completely separate from changing the data. once you print something, the text that you printed is on your screen; it's nowhere else in your code unless it's also a variable somewhere.

delete func and look at what happens when you do this:

df['foo'] = df['two'] * 3

[burnt pilot]

ok

[burnt pilot]

yeah your idea worked

so should I delete the print('...')

and try it again

[steady basalt]

Are u making a new col or printing in ur terminal

[serene scaffold]

@burnt pilot the solution to your problem does not involve any for loops. you'll need to use df.loc and boolean indexing to create the new column.

which I realize probably sounds intimidating at the moment. I'm concerned that your instruction has not adequately prepared you for this task.

[burnt pilot]

im trying to create a new "kol3"

[serene scaffold]

do print(df['two'] > 1) and see what you get.

[burnt pilot]

0       False
1        True
2       False
3        True
4       False
 
9995    False
9996    False
9997    False
9998    False
9999    False

[serene scaffold]

indeed. this is showing you for which rows the value in the two column is greater than 1. do you see how it was able to do that operation for every row, without a loop?

[steady basalt]

When u make a column is it ever a solution to use print function, never done that

[burnt pilot]

yeah but I think I have to apply a function to it for the sake of what we learned so we can practice it

[steady basalt]

Just == “”

[serene scaffold]

so are you making a new column, or are you printing stuff? or are you making a new column, and then printing that new column?

[burnt pilot]

im making a new column 'kol3' with values "greater" "smaller" "medium" depends on the values of the numbers in column "two"

[serene scaffold]

I have to leave suddenly. but the solution to this problem should not involve apply or any loops. if your instructor is insisting that you have to use either of those, they are probably doing you a disservice.

[steady basalt]

I think he tried to make column values from printing into cells

[serene scaffold]

yes. perhaps you can steer them in the right direction.

[steady basalt]

@burnt pilot [] == string

[serene scaffold]

what does this mean?

[burnt pilot]

you mind giving me a quick example on how you think I should do it before you leave

[steady basalt]

Instead of printing into column set the column value == medium or higher than 0 if the other column meets ur value

It’s quite rare I build my own data frame actually

Another way could be to make a dictionary first

And transform that to a df

That could work with ur loop

Insert into a dict

[burnt pilot]

can I do it with lamda ?

[steady basalt]

I guess

I wudnt cause that’s to map

[serene scaffold]

I've already shut down my desktop

[steady basalt]

Try using that loop but instead of data frames use it to make a dictionary

Don’t use print ever though

That’s to output not build a data

wiat a minute, is it even possible to make a dictionary this way

ur gona have dupe keyts

cant recall how pandas takes in dicts

[steady basalt]

create a new column [kol3] and state that df[kol3] == 'smaller' if df[two] < 2

see if it works

[burnt pilot]

sorry I was afk

yeah lemme try

invalid syntax

can I use mapping

[steady basalt]

hol up

[plush jungle]

how do you guys deploy state of the art models? I've been trying to get stylegan2 and stylegan3 working, but I keep getting errors that no one knows how to solve. and I haven't found a jupiter notebook that works either

[burnt pilot]

can I do smth like ({'kol3':['two']>0})

[plush jungle]

do most people just use virtual environments or something?

[steady basalt]

ull have to google it and it will take u30 mins

i tried to fiddle with a similar thing in notebook but has a syntax error too

maybe make a new column with placeholders and then use np.where or smtn

if i didnt have an exam tomorrow id def sit and fix ur problem

[burnt pilot]

I wish the best of luck my friend, don’t worry about me I’ll find a way

[steady basalt]

thanks

it shud b ok but its quite hard

if u dont fix it by tomorrow 6pm letme know ill help

@burnt pilot {colname : [row1 , row2, row3]}

dict_to dataframe

make a list using ur loop []

shud go into dict then u can make that a df and concat to ur df

[steady basalt]

Always how else

[plush jungle]

idk, I was just downloading the code from github and then doing pip install for whatever module it didn't have

[steady basalt]

Then ur using base env

[plush jungle]

I guess I don't really understand how environments work in python

I have this error:

https://stackoverflow.com/questions/72243440/stylegan3-error-ninja-is-required-to-load-c-extensions

[steady basalt]

How do u launch jupyter

[plush jungle]

do you think setting up a venv would fix it?

[plush jungle]

from chrome

[steady basalt]

I’m not sure what you are installing

I’ve never heard of style Egan

[steady basalt]

How do you do that

Do u have miniforge

I have a effective op method

Get miniforge

Are u Mac

[plush jungle]

It's a state of the art Generative Adversarial Network designed to make realistic human faces

I'm on windows 10

https://github.com/NVlabs/stylegan3

[steady basalt]

Why are u deploying GANS without being able to use python envs 🥲

One step at a time

Do u have conda

[plush jungle]

how do I check from terminal?

[steady basalt]

Download miniforge it’s on GitHub

I use that

[plush jungle]

that's for virutal environments?

[steady basalt]

I guess u can also use mini conda

Or anaconda navigator

[plush jungle]

i'm downloading miniforge3

[steady basalt]

Open cmd when it’s installed properly

[plush jungle]

what exactly is conda?

like what does it do

[steady basalt]

Package manager

[plush jungle]

oh ok

[steady basalt]

Well I like pip

But maybe ur issue is pip didn’t install dependencies

[plush jungle]

oh I see

[steady basalt]

Is ninja require to run ur network

[plush jungle]

yes

[steady basalt]

Ok

Is ur manager installed

[plush jungle]

just about

when it does, what do I put in terminal?

[steady basalt]

Open cmd and say does it say (base)

Then : or smtn

[plush jungle]

ok it's installed

[steady basalt]

Show ur command line

[steady basalt]

Now I use mac that has bash so I don’t need the conda prompt but u might on windows

Can u install anaconda

[plush jungle]

is that different than conda?

[steady basalt]

It uses conda

It’s a channel to download from

When u installed miniforge did u get a app

Or anything

Yeah on windows u need prompt

It’s like cmd but just for conda

[plush jungle]

how do I open that? I don't see a conda app even though it downloaded

[steady basalt]

Ur gona need to actually download anaconda on windows it comes with the prompt app

I think

[plush jungle]

ok

it's downloading

so the idea is that I'll be able to run the code through anaconda?

and it'll install all the dependencies automatically?

[supple herald]

It won't install them automatically, you install them into the venv

[steady basalt]

Yep

Don’t bother with the navigator u don’t need that it’s just bloatware

[supple herald]

I just started learning about venvs today haha

[steady basalt]

U need the prompt

[steady basalt]

Dependencies should be flagged and installed head

Yes

But first we need to check what’s on ur computer

I think u pip installed to ur python folder idk

I don’t know how it works when there’s no env

Done?

[plush jungle]

it's taking a while to download, probably cause i'm on starbucks wifi

it says 10 min left

let me go home real quick and maybe it'll go quicker

[steady basalt]

Did anything happen when u installed mini conda or forge

[plush jungle]

it finished downloading, but I can't find it anywhere now...

[steady basalt]

Downloads folder

[plush jungle]

ok I found it, I ran _conda.exe

and it opened this window

and immediately closed it

I can't get it to stay open

[steady basalt]

Ok just get the conda prompt

[plush jungle]

how

[steady basalt]

Anaconda

[plush jungle]

oh ok

[steady basalt]

Mac 4 life

Or Linux

[plush jungle]

@steady basalt ok finally it's downloaded

[steady basalt]

Install

[plush jungle]

once it finishes installing, I open up an anaconda prompt?

[steady basalt]

Yes

Tho it might be better for u to do this via the gui than terminal

[plush jungle]

ok

[steady basalt]

Anyway if u open the prompt u shud be faced with base env and able to type in it

[plush jungle]

@steady basalt when I run _conda.exe it opens and closes

is that not how you start anaconda?

[steady basalt]

Run prompt

[plush jungle]

prompt?

is there a prompt.exe?

@steady basalt oh I got it, I rand _conda.exe from terminal and it worked

so what now

[misty flint]

if anybody is interested in DS opportunities and comes from non-traditional background, i posted one from twitch in #career-advice

The Twitch x Onramp Data Science Apprenticeship provides candidates with the rigorous skills training and support they need to launch a successful data science career. At Onramp, learning and growth are incorporated directly into the application. We provide training plans to help you prepare for the technical assessment and materials to help you put your best foot forward in the application itself. Did you attend a data science bootcamp, take community college courses, or learn online? This could be the perfect opportunity for you!

[steady basalt]

Ok

Try list

To list ur packages

[plush jungle]

yeah it listed them

how do I run the StyleGan code inside anaconda?

[median moat]

That looks great honestly. If I wasn't such a noob I would definitely apply. Rex hooking people up with resources though 👍

[novel python]

I'm just confused about being able to work for US, since the job would be remote. Even if I'm outside US am I still eligible to work in US or is that for residents/work visa only? Got a little bit confused on that part.

[misty flint]

you cant for this position. mostly because twitch is in an incorporated company.

what does that mean? that means that they need to have offices in the country they are hiring for due to tax reasons

its an HR thing

[misty flint]

i think people from non-traditional backgrounds should def go for apprenticeship programs, since they are tailored for them. i like advocating for diversity in tech, since i believe its important

variety of opinions and backgrounds can help foster new ideas and innovation

[novel python]

oooo ok, makes sense, ty for the explanation!

[safe elk]

Yep ...and moar fun too ... moar viewpoints to discuss in meetings and over coffee

[misty flint]

yeah for sure

interesting coffee talks for sure

inb4 startup ideas

[safe elk]

Drinking coffee rn

[misty flint]

lmao

[novel python]

i just drank one too

Rex, is there a "best approach" for trying to get a remote job without experience, or will remote companies always prioritize people with experience? I come from a Chemical Engineering background, finishing my PhD, but I really found my passion while learning about DS/ML. Did a plenty of courses and I feel pretty confident already to apply/deploy models, but I have literally no experience in the area, and it sounds awkward I guess to most companies if I tell my major is in complete different area. Not sure where to step next from here

sorry for the big question also, feel free to skip it if it's too off topic

[misty flint]

most companies, remote or in-person, tend to prioritize people with experience; its been tough for juniors/entry-level peeps lately (literally me lol). there have been a number of people that have been in a similar position (physics phd, molecular bio, geology, etc.) who are trying to make the switch to tech that talked about their experience in #career-advice . just look at @safe elk - he was a chem grad student before switching lol

someone senior was really helpful in giving advice to phd's. let me see if i can find that thread again.

[novel python]

yeah, entry pos been quite hard to find

[novel python]

tyty

[misty flint]

#career-advice message

#career-advice message this was for FR specifically

so not sure how much applies to where you are located

@novel python tbh you can try to ask in #career-advice too since maybe someone else can speak to your situation. my undergrad was in molecular bio, then i spent some time in healthcare before my current grad program

[novel python]

ty alot! it'll help regardless

[novel python]

what you aiming for rn?

[misty flint]

hmm?

im aiming for any position that will accept me for now

uber competitive

out there for entry levels

[worldly dawn]

this is more like #career-advice type discussion but:

• Companies don't prioritize people with experience or not. They would typically have an idea of the seniority level they are looking for and will put out an ad for it (ex: hiring a junior engineer VS hiring a senior engineer). Obviously, if someone exceptional come along, exceptional results may ensue
• It's easier for entry level engineers to be onsite as it's easier to train them and help them. But that would not preclude remote positions.

[worldly dawn]

People are paid based on their location, not the company's HQ.
So then it depends on:

• If the company is familiar with your country
• If the company already has a tax/legal entity in your country
• What is the usual rate for people with your level of experience in your country
• How remote friendly is the company in general

[novel python]

yeah I went a bit off topic there, sorry for that.

Thanks a lot for the clarification!

[wild pagoda]

Currently my code is like this

            channel_rq_response = requests.request(rq_method, channel_rq_url, headers = rq_headers).json()
            df_channel = pd.DataFrame(channel_rq_response)
            df_channel.drop(columns = ["course_name","driving_env"])
            df_channel.to_csv(channel_file, index=False)


            analysis_rq_response = requests.request(rq_method, analysis_rq_url, headers = rq_headers).json()
            df_analysis = pd.DataFrame(analysis_rq_response)
            df_analysis.drop(columns = ["course_name","driving_env"])  <- error here
            df_analysis.to_csv(analysis_file, index=False)

But it have this error

KeyError: "['course_name', 'driving_env'] not found in axis"

But it does, idk why

[royal crest]

it means those columns with those names don't exist

[wild pagoda]

it does, when i print row in df

[royal crest]

what do you get when you do df_analysis.columns.tolist()

[lilac dagger]

.bm will do!

[wild pagoda]

['course_name', 'driving_env', 'id', 'lat_1', 'lat_2', 'lat_3', 'lat_4', 'lon_1', 'lon_2', 'lon_3', 'lon_4']

[runic zodiac]

Thanks but it didnt work

[lilac dagger]

What's the bare bones simplest neural network I should make after I finish this course?

[delicate apex]

Though I far prefer staying a lurker, my doubt of resolution for the issue caused me to do as you suggested and open an issue (https://github.com/pandas-dev/pandas/issues/47053). Good thing, too, because the primary error is still present in the latest commits.

I did solve the 2nd and 3rd problem, however:

with open('kaboom.html', 'w') as f:
    f.write('bye bye style tag\n')
    dfsty.to_html('kaboom.html')

It turns out that the chain of functions that to_html calls contains a with open as as well, meaning the block above is equivalent to

with open('foo.txt', 'w') as k:
    k.write('first line one\nfirst line two')
    with open('foo.txt', 'w') as f:
        f.write('second write call 1\nsecond write call 2')

The second nested context manager exits first, saving its output to the file, and then the first one exits, overwriting the beginning of the file. Dedenting the dfsty call fixes that issue

As for the other problem: f.write(dfsty.render()) is deprecated, but f.write(dfsty.to_html()) returns a string if called without arguments, and runs fine.

[steady basalt]

did you do it

@sweet venture https://discord.com/channels/267624335836053506/650401909852864553

[golden linden]

Check out this guys it's my first data science project: https://youtu.be/QNBgx-w0tvA

[steady basalt]

Very nice

You gona use big csv of dna strings?

[exotic nebula]

How can I set the black line color on matplotlib?

[steady basalt]

Edges?

Oh label?

[green wasp]

Any data engineer here? I’m reading on it and trying to see if I like it and I’m confused about one thing. Is it a no code position, usually?

Does it share the same role to data analysis and science that sysadmin shares with software developers? I.E boring work that’s mind boggling easy and repetitive?

I can’t find a clear answer, and I understand it varies from company to company

[steady basalt]

It requires much harder code than data science

Apparently

Closer to SWE

[green wasp]

Mmmmh then it might be super interesting

I’ll continue reading up on it!

[haughty topaz]

why is looping over a dataframe a bad thing?

[steady basalt]

It’s quite slow

Try df.iterrows

[unborn crow]

Hey. does anyone know something like codecademy for statistics ?

[serene scaffold]

That's still looping over it

[haughty topaz]

the_100_most_common_words = list(final_dic.keys()) # List of 100 common words 
the_100_most_common_words = [i.lower() for i in the_100_most_common_words]

df_label_vector = pd.DataFrame(columns = ["label", "vector"]) # Create the new dataframe with label & vector

for column_name, column_data in df_spoken_sentences_by_characters.iteritems():
    current_character = column_name.split(" ")[0][:-1].lower() # Saves the current character name
    for item in column_data:
        for sentence in item:
            sentence = sentence.lower()
            tokens = nltk.word_tokenize(sentence)
            this_vector = len(the_100_most_common_words)*[0]
            for common_word in the_100_most_common_words:
                if common_word in tokens:
                    this_vector[the_100_most_common_words.index(common_word)] = 1
            df_label_vector.loc[len(df_label_vector.index)] = [current_character, this_vector]
df_spoken_sentences_by_characters.head(5)
df_label_vector.head(5)

Could someone take a look at this loop? Is this a bad way to go about it?

☝️ This is the df_spoken_sentences_by_characters

[serene scaffold]

Because pandas and numpy are implemented in C, and you want to leverage the optimized C code

@haughty topaz are you just trying to find the n most frequent words in the data?

[haughty topaz]

Yeah, for example this loop takes about 40secs

[haughty topaz]

No

For every sentence the character has spoken I want to get a vector of 100 zeros and when the sentence spoken by a character has a common word in it, the value of that index should be 1

[serene scaffold]

I have to get ready for work, but I think the only way I could wrap my head around the intended behavior in a reasonable amount of time would be to see an example input and output on a small sample.

[jagged summit]

Do u guys know data structure?

I'm struggling

[serene scaffold]

Try #algos-and-data-structs. But remember to ask your actual question. Don't ask to ask

[haughty topaz]

This is the output 👇

[jagged summit]

Okay

[haughty topaz]

Np, I mean it works right now but I believe it can be done more time efficient

[serene scaffold]

Yes, but the only way I will be able to understand what it does is to see an input and output

A small one. As text.

[vagrant monolith]

Hi

I want to turn a simple ANN to an RNN

with connecting one layer to another using keras

I --> A --> B --> C --> O

i want to connect layer C to A in backwards direction

[steady basalt]

u mean. binary trees

[somber prism]

guys i loaded a dataset using tf.data.Dataset.from tensor slices and my map function is like this ```
def preprocessing(x, img_path):
print(dir(x))
name1 = str(x[0].numpy())
name2 = str(x[3].numpy())
num1 = str(x[1].numpy())
num2 = str(x[2].numpy())
target = float(x[4])

    img_name1 = f'{name1}_{add_zeros(num1)}.jpg'
    img_name2 = f'{name2}_{add_zeros(num2)}.jpg'
    
    img1 = plt.imread(os.path.join(img_path, name1, img_name1))
    img2 = plt.imread(os.path.join(img_path, name2, img_name2))
    
    return tf.convert_to_tensor([img1, img2, target])

[unborn crow]

hey, i asked before but was ignored, does anyone know good resources to learn stochastic and statistic ?

[haughty topaz]

School

[unborn crow]

I spent my time in school being an Idiot and taking every drug i could find

i would need something more online

[haughty topaz]

Bruh

Online school

Harvard has a lot of free courses

[unborn crow]

That looks like an actual answer

[lapis sequoia]

book should be fine? I can suggest one good book about stats.

it has stochastic prosses as well yes.

[unborn crow]

if it is in englixh or german, yes

[wooden sail]

the books "fundamentals of statistical signal processing" by steven kay are pretty aight

[unborn crow]

Holy S - that thing is expensive

[wooden sail]

if still in uni, you can borrow it from the library

i'm sure there are free alternatives though, it was just a suggestion

[unborn crow]

sadly i am old and not in uni and have never been

but thanks none the less

[burnt pilot]

hey guys

x = pd.read_csv('ex6.csv')
print('x = \n', x)
print("------------------------------------------------------")

#a
df = x.set_index(['key'])
print('df = \n', df)
print("------------------------------------------------------")

df['one'].plot.line(x="key='A'");

im trying to create this line

from a dataframe

where X is 'A' from column 'key'

and Y its the info from column 'one'

[lapis sequoia]

baron should work for you.

or Ross.

A FIRST COURSE IN PROBABILITY by Sheldon Ross

or

PROBABILITY AND STATISTICS FOR COMPUTER SCIENTISTS by Michael Baron

sorry for caps the book had those

[unborn crow]

Found both as pdf - thanks

[lapis sequoia]

you're welcome.

[unborn crow]

If you ever need an idiot - let me know

[vagrant monolith]

Hi guys

How can i achieve this ?

A is a custom layer that takes both inputs from the input and C layer

[haughty topaz]

import pandas as pd

common_words = ["yes", "no", "monica", "you"]
sentence1 = "yes, monica is your friend"
sentence2 = "monica? No, she is not your friend"

input_data: dict = {"monica": [[sentence1, sentence2]], "joey": [[sentence1, sentence2]]}
input_df = pd.DataFrame(input_data)

output_data: dict = {"label": ["monica", "monica", "joey", "joey"], "vector": [[1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 1.0, 0.0], [1.0, 0.0, 1.0, 0.0], [0.0, 1.0, 1.0, 0.0]]}
output_df = pd.DataFrame(output_data)

This is a shortened version of the problem, @serene scaffold

This is the input and output df 👇

[serene scaffold]

I'm at work but I'll try to look at this when I can. thanks!

[haughty topaz]

Ty!

[misty flint]

ahh some of these transformer models are too big for live inference

well

i guess if you have enough compute resources it doesnt matter

or maybe im just too dumb

i think thats a factor

[serene scaffold]

In [12]: from sklearn.preprocessing import MultiLabelBinarizer
In [13]: mlb = MultiLabelBinarizer().fit(['yes', 'no', 'monica', 'you'])

In [25]: mlb.transform([['yes', 'monica', 'is', 'your', 'friend'], ['monica', 'no', 'she', 'is', 'not', 'your', 'friend']])
Out[25]:
array([[1, 0, 1, 0],
       [1, 1, 0, 0]])

I don't really understand what the structure of your dataframe means, but MultiLabelBinarizer will do what you're trying to do for free.

if you can "flatten" one of the columns (every element of the column a single list[str]), you can pass that column to the MLB

In [31]: s = pd.Series([['yes', 'monica', 'is', 'your', 'friend'], ['monica', 'no', 'she', 'is', 'not', 'your', 'friend']])

In [32]: s
Out[32]:
0             [yes, monica, is, your, friend]
1    [monica, no, she, is, not, your, friend]
dtype: object

In [33]: mlb.transform(s)
Out[33]:
array([[1, 0, 1, 0],
       [1, 1, 0, 0]])

In [35]: s.pipe(mlb.transform)
Out[35]:
array([[1, 0, 1, 0],
       [1, 1, 0, 0]])

same semantics

[woven rain]

What is an easy way to translate one chapter then use ai for the rest of the book based on my first input

Looking for a sklearn solution

Just which sklearn package not asking for actual code

[tacit basin]

I think transformers would be good for that, but i didn't use transformers too much. https://huggingface.co/tasks/translation

[plain drift]

hi, i'm looking for some guidance on the right way to configure a data loader in pytorch for an array that is too big to have in memory at the same time. this is presumably a common problem, so i'd prefer to use an established approach instead of whatever i might cook up

[woven rain]

thank you

[desert oar]

do you need the entire array loaded at once? you only need to load 1 batch at a time, whatever your batch size is

[wooden sail]

it seems pytorch has torch utils data, where you can find DataLoader. these are nice precisely for these scenarios

[plain drift]

right but what is the best way to load one batch at a time when all the information is stored in one array encoded as a pkl file somewhere

i can convert it into another format but it still seems mysterious how ml people usually deal with this

the example in pytorch's documentation concerns images, which are naturally distributed across many files https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

am i to distribute my array over many files?

[desert oar]

people generally do not store arrays in pkl files

[plain drift]

that's great. hd5 or something?

[desert oar]

sure, or numpy npy/npz files, or something else

what kind of array is this?

[plain drift]

uhh more or less it's a very large stack of stacks of word vectors. estimated around 58GB

[desert oar]

what is the shape of the array

and why did you put a 58 gb array into a pickle file

[plain drift]

i havent made the file yet

its just the code i have

from work im building on

[misty flint]

what did i learn today?

dont try to deploy large language models for inference unless you have the resources

[desert oar]

then choose a file format that lets you read the data in batches!

[plain drift]

I keep seeing conflicting advice. Seems odd that I can't find a quick guide about this.

okay, that seems interesting. i see that there are some memory-mapped data formats.

[misty flint]

many times its not worth the cost

[desert oar]

why do you need memory-mapped data formats? i think you are really overthinking this

what is the shape of the array?

[plain drift]

isn't that what memory mapping is?

i dont know the shape exactly. it's 3d

[desert oar]

that's fine. what do the 3dimensions represent?

[plain drift]

uh...document_index, word_index, embedding_dimension

[desert oar]

okay, so presumably you only need to read a few documents at a time?

[plain drift]

sure, the batch size

[desert oar]

ok, that's what i wanted to clarify

[plain drift]

yeah, i get that there are a lot of ways to do this. i'm just unsure what practitioners usually prefer

i don't want to take a nonstandard approach, act like it's okay, and then look silly in an interview someday or something

that pkl thing is an example

[desert oar]

i don't think there's one standard approach, every problem is a bit different

[plain drift]

if i just do what other people around me are doing, i will be doing a lot of bad things, bc im in academia

[desert oar]

how many documents do you expect to have, roughly?

you can store them in a database for example. postgresql supports arrays, and most databases support binary blobs. that would be good for fast reads in arbitrary batch sizes. 1 doc per row

or you can store 1 document per file, like an image dataset

[plain drift]

seems to initially be 9676 documents. each is kind of long

[desert oar]

or 1 document per array in an hd5 file

[plain drift]

yeah i was considering distributing it across files

[desert oar]

okay, that seems within reason for a database of blobs, or the filesystem, or hdf5

[plain drift]

ok so either option is viable. that's the main thing i was curious about. thanks for walking me through this.

[desert oar]

im not sure if there are practical upper limits on hdf5 file sizes w/ respect to number of arrays

[plain drift]

i think for ease of translation from pytorch's documentation, I might try the one-document-one-file thing.

[wooden sail]

you'd try to make it into a database or make some sort of metadata file that tells you how to walk through smaller files

[desert oar]

you can at least number the files incrementally

the filesystem is basically a specific kind of database

[wooden sail]

yeah

[steady basalt]

@plush jungle did it?

[plush jungle]

I got anaconda working, but I don't know how to use it

how do i run stylegan code from the anaconda terminal

[neat meadow]

Hello, I have a data science related problem, that is to find the number of different trajectories when given a small dataset (.csv) with columns ['x', 'y', 'depth', 'timestamp'], in a dataframe structure with shape (262, 4). You can think of 'x' and 'y' as points to a 2d plot, 'timestamp' is a time number every point 'x', 'y', 'depth' was "detected". Open for any ideas-discussion to implement this, thank you !

[tidal bough]

what's a "trajectory", here?

[neat meadow]

As we "detect" x's and y's, these create some sort of trajectory if you plot them one by one in a 2d scatter plot

For example, if the first 10 points are close together (ex. Euclidean distance), that means there is a 1st trajectory, but if a new point is detected further from the previous that means a new trajectory is starting

[wooden sail]

one way to look at it could be to fit with curves and test when the derivative or curvature surpasses some quantity. it's a sort of clustering based on rate of change, assuming trajectories should be smooth and nice

[neat meadow]

True, it can be thought as clustering, but during further timestamps, some points may overlap starting points, so clustering those together wouldn't be correct

[wooden sail]

right, but they'd only get clustered together if the curve joining them has a small enough derivative

this would be like writing parametric curves whose curvature is bounded, and that's the criterion for "clustering"

[steady basalt]

Open conda prompt

It’s like cmd

[plush jungle]

how do you open the conda prompt?

if I run _conda.exe it opens a prompt but it immediately closes

[neat meadow]

Either through anaconda app by clicking launch in the cmd.exe promt, or search "anaconda prompt"

[neat meadow]

Thanks, I'll have a look on that

[plush jungle]

ok it's open

now I just run the StyleGan code from inside here?

[steady basalt]

U don’t run python there

Does it say (base)

[plush jungle]

yes it does

[steady basalt]

Ok we don’t wana mess ur base env up or u might get annoyed so let’s type conda create ducky

Wait with windows may be different

Get the commands help

What happens if u type

Conda list

[plush jungle]

I don't see create, but there's build

[steady basalt]

Type conda list

[plush jungle]

conda list gives all my modules

[steady basalt]

Show me the command helps

Let’s list your environments

[plush jungle]

what do I type

[steady basalt]

Bring help up

Post that

[plush jungle]

usage: conda-script.py [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

Options:

positional arguments:
  command
    clean        Remove unused packages and caches.
    compare      Compare packages between conda environments.
    config       Modify configuration values in .condarc. This is modeled after the git config command. Writes to the
                 user .condarc file (C:\Users\Alex Busch\.condarc) by default.
    create       Create a new conda environment from a list of specified packages.
    help         Displays a list of available conda commands and their help strings.
    info         Display information about current conda install.
    init         Initialize conda for shell interaction. [Experimental]
    install      Installs a list of packages into a specified conda environment.
    list         List linked packages in a conda environment.
    package      Low-level conda package utility. (EXPERIMENTAL)
    remove       Remove a list of packages from a specified conda environment.
    uninstall    Alias for conda remove.
    run          Run an executable in a conda environment.
    search       Search for packages and display associated information. The input is a MatchSpec, a query language
                 for conda packages. See examples below.
    update       Updates conda packages to the latest compatible version.
    upgrade      Alias for conda update.

optional arguments:
  -h, --help     Show this help message and exit.
  -V, --version  Show the conda version number and exit.

conda commands available from other packages:
  build
  content-trust
  convert
  debug
  develop
  env
  index
  inspect
  metapackage
  pack
  render
  repo
  server
  skeleton
  token
  verify```

[steady basalt]

I’m not not laptop

Type help

Is that it

conda help

[plush jungle]

that's what it gives when I type conda help

conda create doesn't seem to work

CondaValueError: The target prefix is the base prefix. Aborting.```

[steady basalt]

Yeah windows is different to bash

Type info

Conda info

Also if u can screenshot so it’s easier to read

Info shudda only show base

Conda env list

Conda create -n ducky

Show what env list says first

And info

@plush jungle

[plush jungle]

@steady basalt ok now what

[steady basalt]

Conda activate ducky

[plush jungle]

oh cool

it changed from base to ducky

[steady basalt]

I’m confused why your environments are in side a stylegan folder

Nevermind

It’s not

Use why it’s there

That’s weird

As hell

Anyway

Conda install stylegan

Or whatever their command is

[plush jungle]

their command? it's a github repository that I downloaded?

so what am I installing?

[steady basalt]

Oh

[plush jungle]

there's an environment.yml file in the project

[steady basalt]

This isn’t a library like that

Wtf

[plush jungle]

it's this

https://github.com/NVlabs/stylegan3

[steady basalt]

Ok

Don’t u need pyrotechnics

Pytorch

It says on GitHub

This runs on pytorch

They recommend that you make an environment just for this

[wooden sail]

they tell you to do this

[steady basalt]

Are you using pretraibed network

Trained

[plush jungle]

yes

[steady basalt]

Oh lmao

[wooden sail]

yeah this brings pretrained networks

[steady basalt]

It’s not even on mac ??

[wooden sail]

btw if you don't have visual studio installed, you should start the download now

depending on the components you need, the download can be like 8 gigs

[steady basalt]

U need cuda toolkit

What gpu do u have

[plush jungle]

geforce gtx 1070

[steady basalt]

U need these

[thin palm]

what's up python gang, I'm in a take home assignment for this interview. Just a few questions I'm hoping some of you guys can help me with.
We're trying to make a machine learning prediction for the salaries of jobs to share to jobseekers. I'm given 3 files:
1.) test_features.csv (to test on our model)
2.) train_features (train our model on the features)
3.) train_salaries.csv (salaries of each jobId)
Should I concatenate 2 and 3 with their jobId's to add the extra column for salaries and then in the end before I make my model I can seperate my features and target values?

[steady basalt]

Also a 1070 won’t cut it according to them

[plush jungle]

seriously?

[steady basalt]

They say 12gb

Try it

[wooden sail]

you'll get a memory error, yeah

[steady basalt]

Ok ducky is our experimental env so we can do anything right

Conda install all the dependacies

[plush jungle]

but I'm not trying to train it

i'm trying to use the pretrained model

[steady basalt]

Conda install the list I send u

Remember the error before was ninja

[wooden sail]

you can try running this on colab, too. maybe you'll have better luck there

[plush jungle]

what file is this dependency list in?

I can't find it

[steady basalt]

Environment

[wooden sail]

https://github.com/NVlabs/stylegan3/blob/main/environment.yml

in there

[steady basalt]

@wooden sail do such files work as actual scripts to install the list into itself as a new env

Or he has to conda install them one at a time

Conda install cudatoolkit

Make it 11.2

11.1*

[wooden sail]

used with conda, yes. the command in the screen shot does precisely that

[steady basalt]

That’s cool but it’s requiring a package that’s inside itself

[wooden sail]

which one

[steady basalt]

Or does conda not have the tool kit

We need to see what happens when you conda install cudatoolkit

[plush jungle]

I did this and it's still the same

[steady basalt]

Pip?

Pip install it

[thin palm]

so would anyone concatenate the two files into one?

[wooden sail]

try conda install -c anaconda cudatoolkit

[steady basalt]

Why don’t u have test salaries

[thin palm]

because the "test salaries" will be my eventual file once I'm done building my model

that's my eventual goal!

[wooden sail]

you shouldn't have to concatenate. features seems to be the input, and salaries the output

[steady basalt]

How do you know if you got the predictions right @thin palm

[thin palm]

You don't know, it's my model I'm confident in as part of the assignment

[wooden sail]

sounds about right

[steady basalt]

The teacher will see that?

[wooden sail]

interviewer, sounds like

[steady basalt]

You’re meant to evaluate that and discuss but I guess they keep it secret

[thin palm]

correct yes, it's for an interview

[steady basalt]

See what error and score u get on the training set

[wooden sail]

depending on how the data looks and how much you have, you could augment or split it to do your own tests ahead of time. but anyway. features are inputs, salaries are presumably outputs, no need to concatenate

[thin palm]

ahh okay so then I'll just set my "Y" (target variable) to that salary column

I just wanted to concatenate to deal with one file but you're right

[steady basalt]

This is how titanic Kaggle users do things is it for a entry level

[brave sand]

what are some applications for multi agent reinforcement learning?

[wooden sail]

mhm. your task description you were given should make it clear which one is the output and which one is the input, but this seems to be the approach

[thin palm]

exactly

[wooden sail]

ducky, did my suggestion work for you?

[steady basalt]

So the main thing is you display that you can process it

And also, select hyper parameters

Provide validation curves

[thin palm]

train_features.csv​: Each row represents meta data for an individual job posting.
The “jobId” column represents a unique identifier for the job posting. The remaining
columns describe features of the job posting.
● train_salaries.csv​ :Each row associates a“jobId”with a“salary”.
● test_features.csv​: Similar to​ train_features.csv​, each row represents
metadata for an individual job posting
The first row of each file contains headers for the columns. Keep in mind that the metadata and salaries have been extracted by our aggregation and parsing systems. As such, it’s possible that the data is dirty (may contain errors).

here's a clearer description

[steady basalt]

Clean it and get a feature set using RFE?

Then go for grid search

[thin palm]

because how will the machine learning model know which job posting gets paid what?

[steady basalt]

Training data

[thin palm]

eventually I'll get there.. as of now I'm just getting started

[steady basalt]

X train Y train teach that

[thin palm]

basics will include data cleaning first but I wanted a approach first

[wooden sail]

right, you have to first match the job features to the salary by ID, probably removing ones that are empty or otherwise corrupt. they'll probably not be in order, and that's why you're given the ID

gotta sort through the data first to match up inputs to outputs before feeding them to a network

[steady basalt]

@plush jungle

Pip work?

[wooden sail]

probably with text fields you'll have to one-hot or something of the sort

[steady basalt]

Neural network for this?

The fuckkkk

Linear regression man

[steady basalt]

How did u get cuda 11.3

Pip?

Pip install cudatoolkit=11.1

[plush jungle]

with this

conda install -c anaconda cudatoolkit

[wooden sail]

go into the yaml file and replace the cudatookit line with cudatoolkit>=11.1

and try again

[thin palm]

that's what I was getting at, it gives me matching jobID's for a reason.. so where would I match these up?

[wooden sail]

you match those up in your code when you generate the training pairs

[thin palm]

I'm confusing myself cause usually the target variable is in the same file as the features and I seperate them at the end

[wooden sail]

you know, input with label

paired by ID

[steady basalt]

@thin palm it’s so funny they make this a problem IRL it’s one csv

For this reason I hate Kaggle as it is counterintuitive for beginners

[thin palm]

exactly! just usually one. So I'm like uhh do I need to concat from the get go?

[wooden sail]

there is no concat at all

[plush jungle]

that fixed it

[wooden sail]

when you train a network, you need an input and an output

this association is needed when you train the network

[steady basalt]

Sklearn your x train to be train features

[thin palm]

sorry @wooden sail I know you mentioned this but how do I pair the jobID's then? it's gonna need to know what each job posting is wrth

[steady basalt]

And y train to be training salary

[wooden sail]

you give it inputs and outputs, and they need to be in the same order

you sort them by ID

[thin palm]

ohhh

okay they're actually already sorted in order thankfully

soo I see what yo mean

[steady basalt]

U don’t even need to do this yet

[wooden sail]

beautiful

k, gonna hit the sack, good luck

[thin palm]

so I checked the CSV's they both have the same ID's in same order for example
file1: 4,5,6
file2: 4,5,6
as well

thanks man

[steady basalt]

That should be your first step

[thin palm]

I believe both files are already matched up

So i see now actually

Don't even have to do anything for that

[steady basalt]

Hope they didn’t sneak some in 49 rows down

[thin palm]

watch them do so lmao

[steady basalt]

Throw the training data into a model

After u remove features that are correlated

Or low impact

[thin palm]

Gotcha, first gonna do some data cleaning and feature engneering to OHE some features

but on columns such as jobposting7549 what the heck do I do with this?

[steady basalt]

Show .head

The table

A screenshot…

[thin palm]

gotcha sorry haha

[steady basalt]

Where’s the problematic column

[thin palm]

the jobid and companyid, I feel like my ML model is not going to be affected by this. Should i still one hot encode this?

[steady basalt]

Can u make a df combinginf the features to the salary and then remove features lowly correlated

To the salary ?

Train and train

Before that even look at any features that are correlated to each other first

[thin palm]

actually just realized our tester file has to match the machine learning model

so I can't drop it

[steady basalt]

For example if column 4 and 5 have high corr remove one

[thin palm]

absolutely, but then for the tester what do we do?

just remove the same right?

[steady basalt]

Don’t worry about that yet

But yeah you’d repeat

[thin palm]

okay gotcha man, what I'll do is just get started on simple data cleaning and get back to you my man

[steady basalt]

It’s actually cheating though

Or is it

[thin palm]

what is?

[steady basalt]

Nah it’s fine

[thin palm]

I mean I could google this easily I just wanted to know what to do with some approaches!

[steady basalt]

I was just thinking about something else

[thin palm]

because my intuition is to for sure drop jobid and company id lol, since I feel if I one hot encode it'll add more features

for sure for sure

[steady basalt]

Well what if those from company 33 get paid triple everyone else

It’s useful?

We are about to find out

That’s what you need to test

[thin palm]

true true, I'm trying to answer questions I haven't even tested yet! Ahh!

[steady basalt]

In this quiz it’s probably not useful

Do a heatmap first

Also deal with missing data

[thin palm]

perfect, exactly what I was thinking

[steady basalt]

what job is this for

im curious what to expect when i graduate

[thin palm]

data science position!

part of the data science position is to be able to showcase some machine learning skills

[steady basalt]

damn wtf

do u know how hard it is to get to that stage?!?!

how did u apply

im guessing this is USA

wana help me put a matrix in echelon form?

[haughty topaz]

from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer().fit(common_words)

sentence3 = ['yes', ',', 'monica', 'is', 'your', 'friend']
sentence4 = ['monica', '?', 'no', ',', 'she', 'is', 'not', 'your', 'friend']

vector = mlb.transform([['yes', 'monica', 'is', 'your', 'friend'], ['monica', 'no', 'she', 'is', 'not', 'your', 'friend']])
print(vector)

I get these two vectors of length 10 with all zero's? ☝️

the common_words list has a length of 4

[serene scaffold]

change the second statement to mlb = MultiLabelBinarizer().fit([common_words]) and keep everything else the same, then run it again.

[haughty topaz]

oh wow 🤦

My bad

[pulsar tinsel]

HI, I am try Hello, I am trying to solve 8 puzzle with BFS, with numpy arrays to store each move of the empty board. The problem is that when I get to a point after applying operators, my numpy array loses its dimension. Can I send the code here?

[serene scaffold]

!code

[arctic wedge]

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

[pulsar tinsel]

import numpy as np

class Nodo:
    def __init__(self,datos,hijos=None):
        self.datos = datos 
        self.hijos = None
        self.padre = None
        self.coste = None
    
    def set_datos(self,datos):
        self.datos = datos
    
    def get_datos(self):
        return self.datos
    

    '''
    move(operator)->matrix
    operator: 1 is LEFT
              2 is UP
              3 is RIGHT
              4 is DOWN
    '''
    ```

 def mover(self,operator:int): 
        fila = 0 
        col = 0
        #child = np.zeros([3,3])
        print(self.datos)
        child = np.zeros((3,3))
        child = np.copy(self.datos)
        
        print("The array dimension is ", len(child.shape))
        #print(f'This is first copy {child}')
        child.reshape((3,3))
        if operator == 1:
            for i in range(3):
                for j in range(3):
                    if child[i,j] == 0:
                        fila = i
                        col = j
            if col != 0:            
                child[fila,col] = child[fila,col-1]
                child[fila,col-1] = 0
                return child
        
        if operator == 2:
            for i in range(3):
                for j in range(3):
                    if child[i,j] == 0:
                        fila = i
                        col = j
            if fila != 0:
                child[fila,col] = child[fila-1,col]
                child[fila-1,col] = 0
                return child
        
        if operator == 3:
            for i in range(3):
                for j in range(3):
                    if child[i,j] == 0:
                        fila = i
                        col = j
            if col != 2:
                child[fila,col] = child[fila,col+1]
                child[fila,col+1] = 0
                return child

        if operator == 4:
            for i in range(3):
                for j in range(3):
                    if child[i,j] == 0:
                        fila = i
                        col = j
            if fila != 2:
                child[fila,col] = child[fila+1,col]
                child[fila+1,col] = 0
                return child
        
           
    def __str__(self) -> str:
        return str(self.get_datos())

def buscar_solucion_BFS(estado_inicial,solucion):
    solucionado = False
    nodos_visitados=[]
    nodos_frontera = [] 

    nodoInicial = Nodo(estado_inicial)
    
    nodos_frontera.append(nodoInicial)
    while (not solucionado) and len(nodos_frontera) != 0: 
        nodo=nodos_frontera.pop(0) #complejidad_e O(b^d)
        nodos_visitados.append(nodo) 
        if np.array_equal(nodo.get_datos(),solucion):
            solucionado = True
            return nodo 
        else:
            #dato_nodo = nodo.get_datos()

            #Op right
            hijo = nodo.mover(3)
            hijo_izquierdo = Nodo(hijo)
            nodos_frontera.append(hijo_izquierdo)

            #Op down
            hijo =nodo.mover(4)
            hijo_central = Nodo(hijo)
            nodos_frontera.append(hijo_central)

            #Op left
            hijo = nodo.mover(1)
            hijo_der = Nodo(hijo)
            nodos_frontera.append(hijo_der)

            #Op up
            hijo = nodo.mover(2)
            hijo_der = Nodo(hijo)
            nodos_frontera.append(hijo_der)

if __name__ == "__main__":
    estado_inicial = np.array([[1,0,2],
                               [4,5,3],
                               [7,8,6]])
    
    solucion = np.array([[1,2,3],
                         [4,5,6],
                         [7,8,0]])
    nodo_solucion = buscar_solucion_BFS(estado_inicial,solucion)
    resultado = []
    nodo = nodo_solucion
    while nodo.get_padre() != None:  #Backtracking hasta llegar al nodo padre.
        resultado.append(nodo.get_datos())
        nodo = nodo.get_padre()

    #resultado.append(estado_inicial)
    resultado.reverse() 
    print(resultado)

This is my error.

Sorry for sending the code in parts. The 'move()' method is part of the Node class.

[tidal bough]

child = np.copy(self.datos), so perhaps datos shrinks in size?

[pulsar tinsel]

Yes, somewhere they lose their dimension. But I still don't know why.

I have tried to avoid the copy() method, copying data by data from 'self.data' to 'child'. But it keeps happening.

[late peak]

anybody real familiar with pandas?

[serene scaffold]

you have to ask your actual question

[late peak]

@serene scaffoldI posted a question in #☕help-coffee if you're familiar with pandas

[serene scaffold]

next time, just do "pandas question in #☕help-coffee", instead of directing your question to a specific person.

[late peak]

👌

[eager wedge]

effnet = tf.keras.applications.EfficientNetB0(
    include_top=False,
    weights='imagenet',
    input_shape=(i_size,i_size,3),
    pooling=None,
    classifier_activation="softmax"
)
model = effnet.output
model = layers.Dense(3, activation='softmax')(model)
model = models.Model(inputs=effnet.input, outputs = model)

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(x_train,y_train, validation_data=(x_test,y_test), verbose = 1, epochs=75, batch_size=32)

model.summary()

ValueError: Shapes (None, 3) and (None, 7, 7, 3) are incompatible

Why no work? Pls help. It is three class classification

[hasty grail]

The error message describes the problem pretty clearly

[eager wedge]

Why is it 7,7,3

I used softmax with 3 neurons

[hasty grail]

basically the output shape of the previous layer does not match the expected input shape of the next layer

[eager wedge]

how would this happen?

[hasty grail]

you should check the output shape of EfficientNetB0

[eager wedge]

ok thx

[viscid flume]

Hi, I forked this repo (https://github.com/TachibanaYoshino/AnimeGANv2) for my own use, and when I tried to use the wgan implementations in there, it'd keep erroring out. I'm currently trying to fix it (for 2 days now), and have had a bit of progress, may I send the code?

[brave sand]

what are some applications for MARL?

[hasty grail]

don't ask, just provide as much information as possible so others can better decide whether they can help you

[viscid flume]

Ok

[arctic wedge]

Hey @viscid flume!

You either uploaded a .txt file or entered a message that was too long. Please use our paste bin instead.

[viscid flume]

https://paste.pythondiscord.com/ibodocewet

#discrimator.py
from tools.ops import *
from math import log10 as log

@tf.function(experimental_compile=True)
def D_net(x_init, ch, n_dis, sn, reuse, scope):
    channel = ch // 2
    with tf.variable_scope(scope, reuse=reuse):
        x = conv(x_init, channel, kernel=3, stride=1, pad=1, use_bias=False, sn=sn, scope='conv_0')
        x = tf.cast(x,float)
        x = lrelu(x, 0.2)
        i = 1

        while i < n_dis:
            x = conv(x, channel * 2, kernel=3, stride=2, pad=1, use_bias=False, sn=sn, scope='conv_s2_' + str(i))
            x = lrelu(x, 0.2)

            x = conv(x, channel * 4, kernel=3, stride=1, pad=1, use_bias=False, sn=sn, scope='conv_s1_' + str(i))
            x = layer_norm(x, scope='1_norm_' + str(i))
            x = lrelu(x, 0.2)

            channel = channel * 2
            i += 1
        
        x = tf.nn.dropout(x, rate=0.55, seed = 1)
        x = conv(x, channel * 2, kernel=3, stride=1, pad=1, use_bias=False, sn=sn, scope='last_conv')
        x = layer_norm(x, scope='2_ins_norm')
        x = lrelu(x, 0.2)

        x = conv(x, channels=1, kernel=3, stride=1, pad=1, use_bias=False, sn=sn, scope='D_logit')
        return x```

[pulsar tinsel]

I think I know what the error is. The bug is that when there is no movement up, it returns a None, and this cannot be traversed when it pops up. But I don't know how to fix it yet, what should I return when I don't create the child node?

[viscid flume]

https://paste.pythondiscord.com/lapabasizi

https://paste.pythondiscord.com/foqisupaci

#VSCode Error output:
    /home/allen623329/AnimeGANv2/train.py:95 main  *
        tf.function(AnimeGANv2(sess, args).build_model(AnimeGANv2(sess, args)), experimental_compile=True)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    /home/allen623329/AnimeGANv2/AnimeGANv2.py:163 build_model  *
        d_loss = int(self.d_adv_weight) * discriminator_loss(self.gan_type, anime_logit, anime_gray_logit, generated_logit, smooth_logit) + self.gradient_penalty(real=self.anime, fake=self.generated)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    /home/allen623329/AnimeGANv2/AnimeGANv2.py:122 gradient_penalty  *
        logit, _= tf.function(d,experimental_compile=True)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:1069 function
        return decorated(func)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:1058 decorated
        Function(
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:302 __init__
        self._function_spec = function_lib.FunctionSpec.from_function_and_signature(
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/function.py:1433 from_function_and_signature
        fullargspec = tf_inspect.getfullargspec(python_function)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/util/tf_inspect.py:257 getfullargspec
        return _getfullargspec(target)
    /usr/lib/python3.8/inspect.py:1144 getfullargspec
        raise TypeError('unsupported callable') from ex
    TypeError: unsupported callable```

[viscid flume]

Environment: python 3.8 venv, Gentoo, nvidia-tensorflow-1.15.5 (tried converting to tensorflow 2, broken)

that's all, ig

[hasty grail]

hmm maybe you can drop the tf.function decorator if that's possible

[viscid flume]

from?

[hasty grail]

the function you posted in the code

[viscid flume]

I'll try

Exception has occurred: OperatorNotAllowedInGraphError
in converted code:
    relative to /home/allen623329:

    AnimeGANv2/train.py:95 main  *
        tf.function(AnimeGANv2(sess, args).build_model(AnimeGANv2(sess, args)), experimental_compile=True)
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    AnimeGANv2/AnimeGANv2.py:162 build_model  *
        d_loss = int(self.d_adv_weight) * discriminator_loss(self.gan_type, anime_logit, anime_gray_logit, generated_logit, smooth_logit) + self.gradient_penalty(real=self.anime, fake=self.generated)
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    AnimeGANv2/AnimeGANv2.py:121 gradient_penalty  *
        logit, _= d1()
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/framework/ops.py:547 __iter__
        self._disallow_iteration()
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/framework/ops.py:540 _disallow_iteration
        self._disallow_when_autograph_enabled("iterating over `tf.Tensor`")
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/framework/ops.py:516 _disallow_when_autograph_enabled
        raise errors.OperatorNotAllowedInGraphError(

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.```

[hasty grail]

ok so seems that you need to use tf.function for this

try removing experimental_compile

[viscid flume]

Ok

[hasty grail]

so your function just becomes

@tf.function
def D_net(x_init, ch, n_dis, sn, reuse, scope):
    ...

[viscid flume]

ok

in converted code:

    /home/allen623329/AnimeGANv2/train.py:95 main  *
        tf.function(AnimeGANv2(sess, args).build_model(AnimeGANv2(sess, args)))
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    /home/allen623329/AnimeGANv2/AnimeGANv2.py:162 build_model  *
        d_loss = int(self.d_adv_weight) * discriminator_loss(self.gan_type, anime_logit, anime_gray_logit, generated_logit, smooth_logit) + self.gradient_penalty(real=self.anime, fake=self.generated)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    /home/allen623329/AnimeGANv2/AnimeGANv2.py:121 gradient_penalty  *
        logit, _= tf.function(d)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:1069 function
        return decorated(func)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:1058 decorated
        Function(
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:302 __init__
        self._function_spec = function_lib.FunctionSpec.from_function_and_signature(
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/function.py:1433 from_function_and_signature
        fullargspec = tf_inspect.getfullargspec(python_function)
    /home/allen623329/modeltraining/lib/python3.8/site-packages/tensorflow_core/python/util/tf_inspect.py:257 getfullargspec
        return _getfullargspec(target)
    /usr/lib/python3.8/inspect.py:1144 getfullargspec
        raise TypeError('unsupported callable') from ex

    TypeError: unsupported callable```

This, however

    def gradient_penalty(self, real, fake):
        if self.gan_type.__contains__('dragan') :
            eps = tf.random.uniform(shape=tf.shape(real), minval=0., maxval=1.)
            _, x_var = tf.nn.moments(real, axes=[0, 1, 2, 3])
            x_std = tf.sqrt(x_var)  # magnitude of noise decides the size of local region

            fake = real + 0.5 * x_std * eps

        alpha = tf.random.uniform(shape=[self.batch_size, 1, 1, 1], minval=0., maxval=1.)
        interpolated = real + alpha * (fake - real)
        interpo = tf.expand_dims(interpolated,0)

        def d1(inter):
            return self.discriminator(inter,reuse=True)
        logit, _= tf.vectorized_map(d1,elems=interpo)

        grad = tf.gradients(logit, interpolated)[0] # gradient of D(interpolated)
        grad_norm = tf.norm(flatten(grad), axis=1) # l2 norm

        # WGAN - LP
        if self.gan_type.__contains__('lp'):
            return self.ld * tf.reduce_mean(tf.square(tf.maximum(0.0, grad_norm - 1.)))

        elif self.gan_type.__contains__('gp') or self.gan_type == 'dragan' :
            return self.ld * tf.reduce_mean(tf.square(grad_norm - 1.))```
gives me:

in converted code:
    relative to /home/allen623329:

    AnimeGANv2/train.py:95 main  *
        tf.function(AnimeGANv2(sess, args).build_model(AnimeGANv2(sess, args)))
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    AnimeGANv2/AnimeGANv2.py:160 build_model  *
        d_loss = int(self.d_adv_weight) * discriminator_loss(self.gan_type, anime_logit, anime_gray_logit, generated_logit, smooth_logit) + self.gradient_penalty(real=self.anime, fake=self.generated)
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/eager/def_function.py:449 __call__
        self._initialize(args, kwds, add_initializers_to=initializer_map)
    AnimeGANv2/AnimeGANv2.py:121 gradient_penalty  *
        logit, _= tf.vectorized_map(d1,elems=interpo)
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:390 vectorized_map
        return pfor(loop_fn, batch_size)
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:3160 f  *
        return tuple([converter._convert_helper(x).t
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:1406 _convert_helper
        new_outputs = converter(_PforInput(self, y_op, converted_inputs))
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:3160 f  *
        return tuple([converter._convert_helper(x).t
    modeltraining/lib/python3.8/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:1398 _convert_helper
        raise ValueError(

    ValueError: No converter defined for LeakyRelu
    name: "discriminator/LeakyRelu"
    op: "LeakyRelu"
    input: "discriminator/Cast"
    attr {
      key: "T"
      value {
        type: DT_FLOAT
      }
    }
    attr {
      key: "alpha"
      value {
        f: 0.20000000298023224
      }
    }
    
    inputs: [WrappedTensor(t=<tf.Tensor 'discriminator/Cast/pfor/Cast:0' shape=(?, ?, 256, 256, 32) dtype=float32>, is_stacked=True, is_sparse_stacked=False)]. 
    Either add a converter or set --op_conversion_fallback_to_while_loop=True, which may run slower```

[viscid flume]

I wonder if the above is any easier to solve, for most people just solve it by upgrading to TensorFlow 2.3+ and call it a day

[hasty grail]

perhaps you should upgrade it then

it may be that the model you're using is not supported by older versions

[viscid flume]

Last time I checked, it broke, the lsgan used in there diverged

That's why I came back to tf1

¯_(ツ)_/¯

[hasty grail]

are you sure it's not because you were using bad hyperparameters?

[viscid flume]

I don't think so; it was more stable with the same hyperparameters on tf 1 than tf 2

[hasty grail]

wait, so you are able to run the model even on TF2?

[viscid flume]

lsgan

wgan, unsure

but lsgan keeps diverging

Best record: G_loss: D_loss ~ 710:250, tf1, epoch 42 (starting from 0)

[viscid flume]

The original was run on tf 1.15.0
So to speak, should I give tf 2 another shot?

[spare briar]

fix your technical debt, move to torch

tf1 what a nightmare

[royal crest]

context?

[viscid flume]

I'm just trying to use the project

[spare briar]

you need a wgan implementation?

plenty in torch

[viscid flume]

There is a pytorch fork of this, but I don't know if there is wgan in there

Lemme check

[spare briar]

what i mean is

you dont need anime branding to use a style transfer model

[viscid flume]

Okay

[spare briar]

fyi the og wasserstein gan implementation is in torch https://github.com/martinarjovsky/WassersteinGAN

[viscid flume]

What does og even mean?

[spare briar]

the original

[viscid flume]

Okay......
Any wgan-gp or wgan-lp implementations that are recomended?

Nvm

[lilac dagger]

https://paste.pythondiscord.com/kiluwerowi can this be considered a neural network?

[wooden sail]

only loosely. for one, there are no trainable parameters and no nonlinear activation function. these are the important things. this is just a dot product

[lilac dagger]

what do i do? make some logic to decide the weights correctly and not just random.uniform

[wooden sail]

you need optimization for that. some multivariate calculus

[fleet musk]

helo friends. so i started a course on data science
it asked me to set up a environment on anaconda and they provided a requirements.txt file

while trying to set up, i get following error

clicking on Learn More, redirecte me to

https://conda.io/projects/conda/en/latest/user-guide/troubleshooting.html#unsatisfiable

what do?

😦

getting this on jupyter notebook when launched from new environment

[lilac dagger]

you got a course you reccomend to learn neural nets?

[wooden sail]

not really. especially if you mean to do it all by hand as you're doing, i'd recommend instead to study linear algebra, statistics, and multivar calc, since you will need all of them

[viscid flume]

I keep running into this error

[spare briar]

your gpu does not have enough vram

reduce batch size or get a bigger gpu

[viscid flume]

There's 1GB free, and it reports 35MB?

[viscid flume]

I'm on a laptop

[spare briar]

google colab

vast.ai

aws

or the easier solution is reduce batch size

[viscid flume]

I'm a high school student

[fleet musk]

school students shouldnt get high

[wooden sail]

the options anokhi gave you are the easiest fixes. run it on kaggle or colab or something instead, or reduce your batch size

[viscid flume]

Okay

[spare briar]

potentially try gradient accumulation if your batch is too small

[viscid flume]

I don't have access to google colab through my school account

[wooden sail]

it's free though, you could use it with your own account

[viscid flume]

You mean like:

                ((epoch_current - 1) % test_period == 0 and epoch_current != test_epoch and epoch_current > 1):
            # 最后一个epoch训练完成后做正确显示
            epoch_current_show = epoch_current if iteration == (max_iter - 1) else (epoch_current - 1)
            if evaluators:
                model_generator.eval()
                for evaluator in evaluators:
                    result = evaluator.do_inference(model_generator, epoch_current)
                    # 只有主线程返回
                    if result:
                        # 用于解析日志标志
                        logger.info("(*^_^*)")
                        logger.info("Test model at {} dataset at {} epoch".
                                    format(evaluator.data_loader.dataset.__class__.__name__, epoch_current_show))
                    # synchronize after test
                    synchronize()
                model_generator.train()
            test_epoch = epoch_current``` ?

[viscid flume]

Will do

[spare briar]

nope, the idea is you don't call optimizer.step on every batch

[viscid flume]

Okay......

[spare briar]

if you have batch size of 32 and grad accumulate every 2 batches you get effective batch size of 64

since your gpu is small and your batches are forced to be small this may hurt your model performance

so you accumulate gradients to get a larger effective batch size

[wooden sail]

you could use a small amount of momentum, too, but getting the amount right is an artistic endeavor

[spare briar]

you can accumulate grads with or without momentum

these are orthogonal things

[viscid flume]

1660Ti mobile, welp

[spare briar]

you really just shouldnt train models on that gpu

you can get free gpus from colab that are better

[viscid flume]

lol

Uhh, what do I choose in google colab?

[wooden sail]

i wouldn't say they're orthogonal topics, but they do differ in that the parameters are updated immediately per minibatch instead of kept constant until enough batches are accumulated. but anyway, yeah.

wdym what do you choose?