#data-science-and-ml | Python | Page 277

serene scaffold Dec 31, 2020, 6:30 AM

#

    from constants import TEAM_TO_TEAM_ABBR, TEAM_SETS
ModuleNotFoundError: No module named 'constants'```

#

their code contains broken imports.

desert parcel Dec 31, 2020, 6:30 AM

#

What module is he using?

serene scaffold Dec 31, 2020, 6:31 AM

#

https://github.com/vishaalagartha/basketball_reference_scraper

GitHub

vishaalagartha/basketball_reference_scraper

A python module for scraping static and dynamic content from Basketball Reference. - vishaalagartha/basketball_reference_scraper

desert parcel Dec 31, 2020, 6:31 AM

#

that's specific

#

So if you're free now could you help me?

#

It's in #help-carrot

serene scaffold Dec 31, 2020, 6:31 AM

#

not necessarily

desert parcel Dec 31, 2020, 6:31 AM

#

Ahh alright then

serene scaffold Dec 31, 2020, 6:32 AM

#

Sorry about that

sleek fjord Dec 31, 2020, 6:32 AM

#

my friend used that

#

and it worked for him

desert parcel Dec 31, 2020, 6:32 AM

#

serene scaffold Sorry about that

it's fine

sleek fjord Dec 31, 2020, 6:35 AM

#

on the github under issues someone used it 6h ago, with them still being able to use the module

#

https://github.com/vishaalagartha/basketball_reference_scraper/blob/master/setup.py

GitHub

vishaalagartha/basketball_reference_scraper

A python module for scraping static and dynamic content from Basketball Reference. - vishaalagartha/basketball_reference_scraper

#

this is the setup

austere swift Dec 31, 2020, 8:36 AM

#

So i have this distributed data parallel pytorch model that was working perfectly fine earlier, but now whenever I try to run it i get this error
RuntimeError: unable to open shared memory object </torch_4156_4021375431> in read-write mode
on the mp.spawn() line

#

Python 3.8.0, pytorch 1.7.1, cuda 11.0, cudnn 8.0.5, and ubuntu 18.04

#

I did some research into it and the only things that i found was to set the dataloader workers to 0 and to set the max number of segments for shared memory to 8192, neither worked

#

also tried reinstalling cuda, cudnn, and pytorch

#

ik its something in the os since the code didnt change

#

@ me if you have an answer

desert parcel Dec 31, 2020, 9:17 AM

#

what does a shape of (-x, y) mean for a tensor/matrix?

lapis sequoia Dec 31, 2020, 10:06 AM

#

how do you guys prefer to work with enormous excel files even ms access fail to open?

desert parcel Dec 31, 2020, 10:07 AM

#

pass the excel file with pandas

#

then split the file into half or quarters then export it

#

It's just an idea i've actually never done it before

lapis sequoia Dec 31, 2020, 10:10 AM

#

how one could do that? I mean I am a rookie with python I am trying to learn it like how you learn riding a bike

desert parcel Dec 31, 2020, 10:11 AM

#

Oh idk let me mess around and see if I can get it to work

lapis sequoia Dec 31, 2020, 10:11 AM

#

thank you

desert parcel Dec 31, 2020, 10:17 AM

#

@lapis sequoia I got it to work

lapis sequoia Dec 31, 2020, 10:18 AM

#

let me see

desert parcel Dec 31, 2020, 10:20 AM

#

Alright

#

I'll dm you

sullen crescent Dec 31, 2020, 11:40 AM

#

austere swift So i have this distributed data parallel pytorch model that was working perfectl...

I think it's probably because you are running out of shared memory, how much memory does your GPU have? Have you tried rebooting your PC?

austere swift Dec 31, 2020, 11:41 AM

#

sullen crescent I think it's probably because you are running out of shared memory, how much mem...

I’ve rebooted it like 3 times, it’s a server with 384gb system memory and 4 quadro rtx 6000s so 24gb gpu memory each

#

And I’ve run this program multiple times before, it’s just now started doing this

#

So I don’t think it’s a problem with the hardware

sullen crescent Dec 31, 2020, 11:50 AM

#

austere swift I’ve rebooted it like 3 times, it’s a server with 384gb system memory and 4 quad...

have you tried increasing resources with ulimit -n?

austere swift Dec 31, 2020, 11:50 AM

#

I haven’t

#

I can’t try it right now but I’ll try it when I wake up lol

sullen crescent Dec 31, 2020, 11:50 AM

#

With the increase of the model, the deadlock becomes more and more frequent.

austere swift Dec 31, 2020, 11:50 AM

#

Okay, thanks for the help!

sullen crescent Dec 31, 2020, 11:51 AM

#

try ulimit -n 500000

austere swift Dec 31, 2020, 11:51 AM

#

Okay

#

If I have any problems I’ll ping you

green urchin Dec 31, 2020, 12:29 PM

#

I am thinking of learning python. What is the best way to do it?

storm sigil Dec 31, 2020, 12:32 PM

#

!resources @green urchin Hey check this out, we got some cool stuff in here

arctic wedgeBOT Dec 31, 2020, 12:32 PM

#

Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

lapis sequoia Dec 31, 2020, 12:38 PM

#

mmmm i dont wanna sort the output. I mean, maybe i am doing it wrong. I used sklearn LabelEncoder to asign numbers to the classes. And then, after using model.predict, i used np.argmax to get the index with the highest value. So i want 2 things: know if there is something like LabelDecoder, and return the n indexes with highest values

green urchin Dec 31, 2020, 12:39 PM

#

@storm sigil thanks mate

hasty grail Dec 31, 2020, 1:26 PM

#

lapis sequoia mmmm i dont wanna sort the output. I mean, maybe i am doing it wrong. I used skl...

!e

import numpy as np

arr = np.asarray([10, 50, 20, 30, 40])
top_k = arr.argsort()[:3]
print(top_k)

arctic wedgeBOT Dec 31, 2020, 1:26 PM

#

@hasty grail :white_check_mark: Your eval job has completed with return code 0.

[0 2 3]

lapis sequoia Dec 31, 2020, 1:27 PM

#

well, as u can see, thats wrong

#

index 0, which correspond to 10, isnt on the top 3 highest values

hasty grail Dec 31, 2020, 1:27 PM

#

!e

import numpy as np

arr = np.asarray([10, 50, 20, 30, 40])
top_k = arr.argsort()[::-1][:3]
print(top_k)

arctic wedgeBOT Dec 31, 2020, 1:27 PM

#

@hasty grail :white_check_mark: Your eval job has completed with return code 0.

[1 4 3]

hasty grail Dec 31, 2020, 1:27 PM

#

just had to invert it

#

!e

import numpy as np

arr = np.asarray([10, 50, 20, 30, 40])
top_k = arr.argsort()[-3:][::-1]
print(top_k)

arctic wedgeBOT Dec 31, 2020, 1:28 PM

#

@hasty grail :white_check_mark: Your eval job has completed with return code 0.

[1 4 3]

lapis sequoia Dec 31, 2020, 1:29 PM

#

mmmm ty

#

and can i somehow use sklearn to go from interger to the class name?

hasty grail Dec 31, 2020, 1:30 PM

#

If your class names are in a numpy array then you can just index it

lapis sequoia Dec 31, 2020, 1:30 PM

#

int_train_label = le.fit_transform(train_label).astype(np.float32)```

#

Just the other way

lapis sequoia Dec 31, 2020, 1:31 PM

#

hasty grail If your class names are in a numpy array then you can just index it

ah true lel

hasty grail Dec 31, 2020, 1:31 PM

#

!e

import numpy as np

class_names = np.asarray(['a', 'b', 'c', 'd', 'e'])
predictions = np.asarray([10, 50, 20, 30, 40])
top_k_idx = predictions.argsort()[-3:][::-1]
top_k_classes = class_names[top_k_idx]
print(top_k_classes)

arctic wedgeBOT Dec 31, 2020, 1:31 PM

#

@hasty grail :white_check_mark: Your eval job has completed with return code 0.

['b' 'e' 'd']

lapis sequoia Dec 31, 2020, 1:31 PM

#

ty

grizzled hill Dec 31, 2020, 2:28 PM

#

Any good course about solving differential equations in python ?

hollow scarab Dec 31, 2020, 2:38 PM

#

is there a way to make df.iloc dynamic? I have a dataset with x length but that might change next week

#

and if I df.iloc for a bigger dataset then I get an error

#

I have 260rows, and it won'tl let me select more than 260, but if next weeks file will be longer than that will be lost

lapis sequoia Dec 31, 2020, 3:34 PM

#

hollow scarab I have 260rows, and it won'tl let me select more than 260, but if next weeks fil...

Is there a reason for you not to use .loc?

#

What do you want to use .iloc for

hollow scarab Dec 31, 2020, 3:39 PM

#

@lapis sequoia I had to select more ranges, not sure if that works with loc

#

📎 unknown.png

lapis sequoia Dec 31, 2020, 3:49 PM

#

and what does that range represent?

#

if you want a fixed output regardless of row numbers, you could state a def function priorly

hollow scarab Dec 31, 2020, 3:51 PM

#

and 2. row, which is fixed every week, and 6-268 can change to 6-269, 6-270 depending on week @lapis sequoia

#

the column length is also fixed

#

just that 6-268 can be +/- 1-2

golden briar Dec 31, 2020, 4:09 PM

#

Hi

lapis sequoia Dec 31, 2020, 4:12 PM

#

i have downloaded some weights from internet for inceptionv3

#

how can i load them?

fringe gyro Dec 31, 2020, 4:40 PM

#

hello everyone

#

who help plz

#

Choose the first note randomly or consider the first note of the
original score (e.g. G) ;
See in the table the notes that succeed it (Ex: for the note G, the successors
are : G, G i, D, D, C, A) ;
Choose a random note among these successors;
Repeat this process, as many times as necessary, starting from step b) by considering
each time the successors of the last selected note.

#

it's my pb in python

sturdy dune Dec 31, 2020, 5:40 PM

#

2020 has been amazing year for Artificial Intelligence. As it is the last day of 2020, let us quickly recap the top achievements. https://datamahadev.com/top-10-artificial-intelligence-stories-of-2020/

lilac iris Dec 31, 2020, 5:56 PM

#

so i am using imageai to detect stuff from a opencv video feed, problem is i have to record, the analyse, then see it, is there a way to make it run live?

weak sentinel Dec 31, 2020, 6:00 PM

#

Hey, im trying to use a DQN on the LunarLander from OpenAI gym rn and im noticing that everyone else is using a python file instead of a notebook. Why is this? is there some advantage that im missing?

serene scaffold Dec 31, 2020, 6:08 PM

#

sturdy dune 2020 has been amazing year for Artificial Intelligence. As it is the last day o...

you can share your articles if you'd like as long as they're posted in the appropriate channels, as is the case here. Though you might get more satisfying discussion on reddit since comments here are ultimately only seen by a few people.

lapis sequoia Dec 31, 2020, 6:19 PM

#

If i am doing an image classification, and my test images have no back ground, will the nn be able to predict the same images with a background?

gaunt venture Dec 31, 2020, 6:32 PM

#

hello, not sure if i should ask this here or in algos and dat astructs,, im trying to generate all the permutations given 12 numbers, and using between 12 and 6 at a time

#

so if i had [1,3,5,7,9,11,13,15,17,19,21,23],and length 8 i would want for example, ('3', '7', '13', '11', '9', '5', '1', '15')

#

but once i try for length 10,11 or 12, my pc freezes

#

a length of 10 has 239,500,800 permutations, and 11 and 12 lengths have 479,001,600

slender oracle Dec 31, 2020, 6:52 PM

#

The itertools module in the standard library has functions for doing this. e.g. https://docs.python.org/3/library/itertools.html#itertools.permutations

#

Also returns a generator instead of a list, so should be able to avoid any memory issues.

gaunt venture Dec 31, 2020, 7:15 PM

#

ooh thank you! ill test it out, sounds perfect

gaunt venture Dec 31, 2020, 7:47 PM

#

wait did you mean the normal permutations()? that's what i'm using

gentle wagon Dec 31, 2020, 8:25 PM

#

mhm makes sense. Thanks for taking the time to answer my question !!

haughty turtle Dec 31, 2020, 9:55 PM

#

I would need some help because I don't know what algorithm i could use for the following :
- Steve is 25 and he buys everyday orange juice 
- Maria is 23 and she likes to buy smoothies 
Steve & Maria tastes are pretty much the same.

- Juan is 16 and he only drinks sodas 
Juan tastes are not the same as Steve and Maria.

====================================================

I would like to do a matching algorithm that will return the users who have the same drink preference and a close age. To continue with the example, Steve and Maria would be matched together but not Juan. Which one should I use ?```

pseudo rivet Dec 31, 2020, 10:16 PM

#

How do you convert a date string (using Dash's DatePickerSingle) to a date_object that can be used?

harsh reef Dec 31, 2020, 11:13 PM

#

63/150 [===========>..................] - ETA: 1:05 - loss: 5.5540 - rpn_class_loss: 0.1711 - rpn_bbox_loss: 1.3187 - mrcnn_class_loss: 1.6656 - mrcnn_bbox_loss: 0.8899 - mrcnn_mask_loss: 0.6008
ERROR:root:Error processing image {'source': 'suncg', 'path': '/path/to/mlt/8a33bca7ed13c8d2698303625feba21a/000005.png', 'obj_mask_path': '/path/to/node/8a33bca7ed13c8d2698303625feba21a/000005.png', 'cls_mask_path': '/path/to/category/8a33bca7ed13c8d2698303625feba21a/000005.png', 'id': 8067}
Traceback (most recent call last):
ValueError: zero-size array to reduction operation minimum which has no identity

#

hey, can someone help me

#

its showing value error even after i changed the images into json using LabelMe any help?

tranquil nymph Dec 31, 2020, 11:43 PM

#

📎 unknown.png

#

I'm trying to create a panda dataframe from this

#

excel sheet

#

but it's exported from quicken so the first rows are messed up.

#

>>> df = xl.parse("grace_covenant", names=["Split", "Date", "Check #", "Payee", "Category", "Amount", "Balance"], header=7, index_col=1) 
>>> df.head()
Empty DataFrame
Columns: []
Index: []

#

this is my current output

lapis sequoia Dec 31, 2020, 11:54 PM

#

I got a problem. I downloaded a npy file that contains images. Well, i guess images are on RGB. But since i am opening with opencv, i think it takes the images as if they were BGR. How to fix this?

#

cv2.imshow(text[i], cv2.resize(cv2.cvtColor(imgs[i], cv2.COLOR_RGB2BGR), (256,256)))

#

it sucks

#

too complicated

wintry nacelle Jan 1, 2021, 12:05 AM

#

Why does it take it as BGR anyway?

lapis sequoia Jan 1, 2021, 12:05 AM

#

if u know different ways pls tell me

#

ask the developers xD

#

but opencv read images as bgr

wintry nacelle Jan 1, 2021, 12:06 AM

#

Okay I looked it up

#

It's because bgr was the standard for cameras back then

#

So you're stuck doing the conversion

#

If it makes it look less ugly, try spacing the code out between lines

lapis sequoia Jan 1, 2021, 12:07 AM

#

i was wondering if imshow has any argument to fix this

#

because since my img is already an array

#

i cant use imread

wintry nacelle Jan 1, 2021, 12:07 AM

#

I don't think you have any other option

lapis sequoia Jan 1, 2021, 12:07 AM

#

and on imread i can specify how are the image channels

#

https://docs.opencv.org/3.4/d4/da8/group__imgcodecs.html

wintry nacelle Jan 1, 2021, 12:16 AM

#

I am trying to learn machine learning, specifically generative adversarial networks. I am following an example on the Keras website detailing a variational autoencoder (we all gotta start somewhere). While I understand each individual line of code somewhat, I am lost as to what is generally happening, and I don't understand many of the choices made. I put this down to lack of both knowledge and experience. I would like help understanding just what is going on.
Example source: https://keras.io/examples/generative/vae/
Code with comments asking questions: https://paste.pythondiscord.com/wopadonuro.py
Also, one more thing left out. I set the batch size to 4. I wonder if this is a terrible idea.
Current model's progress:

📎 unknown.png

Keras documentation: Variational AutoEncoder

safe tapir Jan 1, 2021, 1:36 AM

#

Is there a way to get jupyter autocomplete for Classes only?

lapis sequoia Jan 1, 2021, 3:54 AM

#

Are there any resources, guides, or books about structuring Python code for scientific applications?

slender oracle Jan 1, 2021, 4:13 AM

#

You could take a look at this: https://drivendata.github.io/cookiecutter-data-science/

Home - Cookiecutter Data Science

A project template and directory structure for Python data science projects.

#

I've used it for one-off/research type things at work

lapis sequoia Jan 1, 2021, 4:22 AM

#

Thanks. I'll give it a look.

fervent flume Jan 1, 2021, 5:52 AM

#

run code in vscode debugger: works fine.
run code from command line: crashes

#

happy new year 🙂

serene scaffold Jan 1, 2021, 6:21 AM

#

fervent flume run code in vscode debugger: works fine. run code from command line: crashes

did you get to the line in the debugger that raised the error?

fervent flume Jan 1, 2021, 6:22 AM

#

it's something to do with pathos.multiprocessing

#

@serene scaffold

#

If I run it one way, everything dies. If i run it another way, it works fine

serene scaffold Jan 1, 2021, 6:23 AM

#

ah. I actually don't know how you'd debug something that's running in parallel. Did you ask in #async-and-concurrency?

fervent flume Jan 1, 2021, 6:23 AM

#

not yet

serene scaffold Jan 1, 2021, 6:23 AM

#

fervent flume not yet

You can ask about anything pertaining to "more than one thing happening at once" there, not strictly async/await stuff.

fervent flume Jan 1, 2021, 6:23 AM

#

thanks

wintry nacelle Jan 1, 2021, 6:34 AM

#

To commemorate the new year, I made something. Result of Keras example code that I shamelessly copied in order to learn more.

📎 index.png

stuck galleon Jan 1, 2021, 6:50 AM

#

Hey guys I’ve been wanting to make a program that would be able to watch a player of a game in a match and by the behavior and habits of that player be able to predict what they would most likely do in future scenarios. Any ideas on where to get started?

lilac iris Jan 1, 2021, 11:17 AM

#

Hey! Im trying to use opencv-python and I have a weird issue, detection on my webcam is perfectly working however when i run it on a file, this happens: https://cdn.discordapp.com/attachments/677243335290912808/794521560597790721/unknown.png

Here is my code

blob = cv2.dnn.blobFromImage(frame, 1/255, (320, 320), (0, 0, 0), 1, crop=False)
    darknet.setInput(blob)
    
    layers = darknet.getLayerNames()
    outputs = [layers[idx[0]-1] for idx in darknet.getUnconnectedOutLayers()]
    outputs = darknet.forward(outputs)
    
    analyse(outputs, frame)

def analyse(outputs, frame):
    img_height, img_width = frame.shape[0], frame.shape[1]
    bbox = []
    class_ids = []
    probabilities = []

    for output in outputs:
        for detection in output:

            class_id = argmax(detection[5:])
            probability = detection[5:][class_id]

            if probability >= 0.5:
                bbox.append([int((detection[0] * img_width)-int(detection[2]*img_width)/2),
                             int((detection[1] * img_width)-int(detection[3]*img_width)/2),
                             int(detection[2]*img_width),
                             int(detection[3]*img_height)])
                class_ids.append(class_id)
                probabilities.append(float(probability))

    indicies = cv2.dnn.NMSBoxes(bbox, probabilities, 0.5, 0.3)
    
    for idx in indicies:
        idx = idx[0]
        box = bbox[idx]
        center_x, center_y, width, height = box

        cv2.rectangle(frame, (center_x, center_y), (center_x+width, center_y+height), (0, 225, 0), 2)
        cv2.putText(frame, f'{classes[class_ids[idx]].title()} - {round(probabilities[idx]*100, 2)}%', (center_x, center_y-10), cv2.FONT_HERSHEY_PLAIN, 1, (0, 225, 0), 2)

sullen crescent Jan 1, 2021, 1:14 PM

#

lilac iris Hey! Im trying to use `opencv-python` and I have a weird issue, detection on my ...

check image scaling parameter on cv2.dnn.blobFromImage

lilac iris Jan 1, 2021, 1:16 PM

#

oh ok

#

i cant rn, ill tell when im free later

lapis sequoia Jan 1, 2021, 1:26 PM

#

wooooooooooo ive discovered something

#

if u fit a model, with 10 epochs, for example, and acc is low

#

and u increase epochs

#

If u dont compile the model again

#

the fit will start from the last epoch

#

idk if i explained xD

#

model.compile

#

Imagine those are diferent cells

#

if ur epochs are 10

#

and u run first cell, and then second

#

model will fit with 10 epochs

#

if u run second cell again

#

it will be as if u train model with 20 epochs

velvet thorn Jan 1, 2021, 1:28 PM

#

yeah

#

because

#

fitting changes the model inplace

lapis sequoia Jan 1, 2021, 1:29 PM

#

xd

#

this guy lmao

arctic wedgeBOT Jan 1, 2021, 1:29 PM

#

@green jungle Per Rule 6, your invite link has been removed. If you believe this was a mistake, please let staff know!

Our server rules can be found here: https://pythondiscord.com/pages/rules

lapis sequoia Jan 1, 2021, 1:30 PM

#

velvet thorn fitting changes the model inplace

yeah didnt know. Like, i was always compiling as long as fitting

lapis sequoia Jan 1, 2021, 2:53 PM

#

What is the best laptop to learn data analysis, machine learning, deep learning and neural networks?

fluid comet Jan 1, 2021, 3:15 PM

#

Hello everyone, please tell me how to get all the elements of the list that pass according to the selection condition
pymongo

barren osprey Jan 1, 2021, 3:18 PM

#

lapis sequoia What is the best laptop to learn data analysis, machine learning, deep learning ...

Umm, there is no "best" for such things. It is a matter of personal preference. Just get a fairly performant laptop, could be a mac could be anything u want, just make sure it's not a slow hag

lapis sequoia Jan 1, 2021, 3:19 PM

#

barren osprey Umm, there is no "best" for such things. It is a matter of personal preference. ...

Thanks

queen stratus Jan 1, 2021, 4:20 PM

#

hey guys, very new in this python thing. just wanted to install tensor flow trying to reproduce results of this https://github.com/NVlabs/noise2noise

GitHub

NVlabs/noise2noise

Noise2Noise: Learning Image Restoration without Clean Data - Official TensorFlow implementation of the ICML 2018 paper - NVlabs/noise2noise

#

📎 unknown.png

#

then this happened

#

it's still going at it

#

wondering if that's normal

mystic steeple Jan 1, 2021, 4:42 PM

#

Hey, so I'm a little bit (a lot bit) of a noob and I have a problem I need to solve. I was wondering if someone could give me some pointers of A) if python can be used to solve this problem and B) if so, some tips for how to get started.
Basically, I have a list of every teaching and learning center in the country. (Roughly 2000) I need to plot where each one is on a map. This would be fairly easy if I had the coordinates for each, but I don't, all I have is the name of the town. And I really do not want to manually get the coordinates for all 2000 centers. 😅
I'm sorry if this is the wrong channel, but any help would be super appreciated.

queen stratus Jan 1, 2021, 4:46 PM

#

I remember a friend of mine doing something similar

serene scaffold Jan 1, 2021, 4:46 PM

#

@mystic steeple google maps probably has an api for what you're trying to do

queen stratus Jan 1, 2021, 4:46 PM

#

I think there is something that auto encodes your stuff in google maps

mystic steeple Jan 1, 2021, 4:47 PM

#

Darn that's exactly what I'm looking for! Thank you, I'll dig around for that.

feral spoke Jan 1, 2021, 5:03 PM

#

Guys I was going through the lecture on stats and at some point they define the formulae for median of grouped data.
Can anyone here explain me how they came across the formulae?

#

Median = L + ((N/2 - Cf)/f) * W

#

where L is the lower limit of median class

#

N is number of observations

#

Cf is cummulative frequency of the previous median class

#

f is the frequency of median class

#

W is the width of median class

#

Kindly ping me if you guys know the solution to it

trim oar Jan 1, 2021, 6:05 PM

#

queen stratus hey guys, very new in this python thing. just wanted to install tensor flow tryi...

If you didn’t have those packages it’s normal. It takes time to install conda or tensor flow

lapis sequoia Jan 1, 2021, 7:14 PM

#

how may i determine how many epochs do i need and what batch size?

stray owl Jan 1, 2021, 7:28 PM

#

lapis sequoia how may i determine how many epochs do i need and what batch size?

"Grip it and rip it" - John Daly, on how many epochs and batch sizes, probably.

#

There isn't a set number, you want to be running your project multiple times to see what works best. Guard against not enough testing data and over-fitting.

lapis sequoia Jan 1, 2021, 7:47 PM

#

stray owl There isn't a set number, you want to be running your project multiple times to ...

how can i know if i am doing this or not XD

#

cuz wow

#

93% acc

bronze skiff Jan 1, 2021, 7:49 PM

#

though you probably want to hypwrparameter tune on a validation set

#

to prevent data leakage

lapis sequoia Jan 1, 2021, 7:50 PM

#

loss: 0.2347 - accuracy: 0.9299 - val_loss: 0.2966 - val_accuracy: 0.9311

lapis sequoia Jan 1, 2021, 7:50 PM

#

bronze skiff though you probably want to hypwrparameter tune on a validation set

what does this mean?

#

train = (34970, 898) validation = (8271, 898)

#

im also using data augmentation

bronze skiff Jan 1, 2021, 8:12 PM

#

hyperparameter tuning on your final eval set poisons your results

lapis sequoia Jan 1, 2021, 8:20 PM

#

what?

trim oar Jan 1, 2021, 8:22 PM

#

lapis sequoia how may i determine how many epochs do i need and what batch size?

Only as many as necessary. Start small. Deploy EarlyStopping to prevent overfitting, and learning scheduler to decay learning rate. Only need to increase it if early stopping didn't stop training and could obviously still learn better

trim oar Jan 1, 2021, 8:25 PM

#

lapis sequoia how can i know if i am doing this or not XD

Is it a classification problem? Becuase if it is, just look at the confusion matrix

lapis sequoia Jan 1, 2021, 8:27 PM

#

okey step by step @trim oar

#

I have 35k images for training and 8k for validation (without counting data augmentation on train)

#

I have 898 classes. 15 epochs and batch size was 32. Model achieved 93% acc.
However, all the images from training/validation havent brackground. So when i tried to predict an image with background, the correct answer isnt even on the top 5

lethal solstice Jan 1, 2021, 8:43 PM

#

Hello everyone

#

someone have an issue with Scrapy like me when you try to do :

class BookingSpider(scrapy.Spider):
    name="booking", 
    start_urls=['https://www.booking.com/']
    
#     Callback function that will be called when starting your spider
    def parse(self, response):
        yield scrapy.FormRequest.from_response(response,
                                               formdata = {'ss': 'Lyon'},
                                               callback = self.after_search)

#

i got each time this error :

2021-01-01 20:40:23 [scrapy.core.engine] INFO: Spider opened
2021-01-01 20:40:23 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.open_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7f89db07af10>>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/scrapy/utils/defer.py", line 157, in maybeDeferred_coro
    result = f(*args, **kw)
  File "/home/jovyan/.local/lib/python3.8/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/opt/conda/lib/python3.8/site-packages/scrapy/extensions/feedexport.py", line 291, in open_spider
    uri_params = self._get_uri_params(spider, feed_options['uri_params'])
  File "/opt/conda/lib/python3.8/site-packages/scrapy/extensions/feedexport.py", line 473, in _get_uri_params
    params[k] = getattr(spider, k)
  File "/opt/conda/lib/python3.8/site-packages/scrapy/spiders/__init__.py", line 36, in logger
    logger = logging.getLogger(self.name)
  File "/opt/conda/lib/python3.8/logging/__init__.py", line 2013, in getLogger
    return Logger.manager.getLogger(name)
  File "/opt/conda/lib/python3.8/logging/__init__.py", line 1285, in getLogger
    raise TypeError('A logger name must be a string')
TypeError: A logger name must be a string

lapis sequoia Jan 1, 2021, 9:30 PM

#

anyone with high knownledges of ML could help me please? I have been stuck for a few days with this

wintry nacelle Jan 1, 2021, 9:36 PM

#

Welp I just installed CUDA so I can use my GPU for training

lapis sequoia Jan 1, 2021, 9:37 PM

#

u can use google colab to use gpu for training

wintry nacelle Jan 1, 2021, 9:37 PM

#

Google colab is slower than running Jupyter natively and always disconnects

#

For me at least

#

I'd just rather use my own equipment'

lapis sequoia Jan 1, 2021, 9:38 PM

#

for me it is working fine

wintry nacelle Jan 1, 2021, 9:38 PM

#

Anyway I'll try to tackle your error while I'm waiting

lapis sequoia Jan 1, 2021, 9:38 PM

#

it is not mine 😛

wintry nacelle Jan 1, 2021, 9:38 PM

#

ok

#

Wait then what was this about:

anyone with high knownledges of ML could help me please? I have been stuck for a few days with this

lapis sequoia Jan 1, 2021, 9:39 PM

#

aaah

#

that one hahaha

wintry nacelle Jan 1, 2021, 9:39 PM

#

I'm generally stuck to, I don't know what z_mean is

lapis sequoia Jan 1, 2021, 9:39 PM

#

yes, that one yes

#

what

wintry nacelle Jan 1, 2021, 9:39 PM

#

One of Keras' examples for training a VEA uses z_mean as a variable

#

I don't know wtf it does

#

Hold on I'll scroll up and fetch it

#

#help-pancakes message

lapis sequoia Jan 1, 2021, 9:42 PM

#

i dont think i can help u xd u better wait @ gm or someone else

trim oar Jan 1, 2021, 9:43 PM

#

lapis sequoia I have 898 classes. 15 epochs and batch size was 32. Model achieved 93% acc. How...

But what was your confusion matrix like

lapis sequoia Jan 1, 2021, 9:43 PM

#

how can i print that?

wintry nacelle Jan 1, 2021, 9:44 PM

#

@ gm or someone else
?

trim oar Jan 1, 2021, 9:44 PM

#

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html

#

Here

lapis sequoia Jan 1, 2021, 9:44 PM

#

okey i will ping u, cant rn

trim oar Jan 1, 2021, 9:44 PM

#

Np

#

I'm also here and there so

wintry nacelle Jan 1, 2021, 9:45 PM

#

Hey ian yu can you help me with my problem?

trim oar Jan 1, 2021, 9:52 PM

#

@wintry nacelle Sorry, I'm not exactly familiar with z means

wintry nacelle Jan 1, 2021, 9:52 PM

#

There are a handful more questions generally related to convolution in my question

#

Such as the numbers set for stride and output layers

#

What CUDA version is required for the latest nightly build?

#

nvm it's 11.0

lapis sequoia Jan 1, 2021, 10:08 PM

#

@trim oar sorry i will have to train the model again q.q. But... i could share u my dataset, the one i use to train the model, and the images i am trying to predict, so maybe u can see what could go wrong

#

but i think is the background from images

trim oar Jan 1, 2021, 10:13 PM

#

Most probably is

wintry nacelle Jan 1, 2021, 10:22 PM

#

Finally I got tensorflow to work with my cpu

#

Only took an hour but man that is an uptick in performance

serene scaffold Jan 1, 2021, 11:44 PM

#

Anyone who's worked with having lots of arrays that may or may not need to be in live memory at any given time: have you had success with storing arrays in a database of some kind? I'm wanting to get started on a Discord bot that will need to store ten or so (400,)-shaped arrays per text channel that it can see.

slender oracle Jan 1, 2021, 11:49 PM

#

serene scaffold Anyone who's worked with having lots of arrays that may or may not need to be in...

Have you decided which database you want to use? I store arrays in tables using both spark and presto. In my case, the underlying storage layer for each is S3 and parquet or ORC files.

#

I think postgres supports arrays too, but haven’t used it in several years.

serene scaffold Jan 2, 2021, 12:15 AM

#

slender oracle Have you decided which database you want to use? I store arrays in tables using ...

I don't know enough about databases to answer that question. MySQL and SQLite were the only ones we discussed in the one database class I took.

slender oracle Jan 2, 2021, 12:19 AM

#

serene scaffold I don't know enough about databases to answer that question. MySQL and SQLite we...

Looks like neither of those support array types, but postgres does. Just double checked: https://www.postgresql.org/docs/9.1/arrays.html

PostgreSQL Documentation

Arrays

serene scaffold Jan 2, 2021, 12:22 AM

#

slender oracle Looks like neither of those support array types, but postgres does. Just double ...

and I could presumably use a postgres-backed database with a database library of my choosing?

slender oracle Jan 2, 2021, 12:26 AM

#

serene scaffold and I could presumably use a postgres-backed database with a database library of...

Yep. psycopg2 is a common one and what we use at work.

#

Though that's sort of because we also use airflow, which requires psycopg2 for postgres connectors.

#

https://github.com/psycopg/psycopg2

lapis sequoia Jan 2, 2021, 2:15 AM

#

can someone help me on nickel pls?

blazing bridge Jan 2, 2021, 5:37 AM

#

Hi guys, I am building a pc, the specs are as follows:

i9 9900k,
Gigabyte Eagle 3070,
16 gb of 3000mhz ram at CL 15,
MSI MAG 360R cooler

Is this a good pc for deep learning

I can specify more parts such as the case and power supply but they aren't as important as the stuff above

onyx monolith Jan 2, 2021, 5:50 AM

#

does anyone here know anything about how to preprocess fMRI datasets for ML?

lapis sequoia Jan 2, 2021, 6:30 AM

#

@blazing bridge your 3070 will be fine for deep learning 👍🏼

blazing bridge Jan 2, 2021, 7:05 AM

#

ok thank you @lapis sequoia

wintry nacelle Jan 2, 2021, 7:11 AM

#

Are there any resources or guides on best ML practices? I feel like I know Tensorflow and Keras enough to finally understand code, but I want to know how to best create/train models

near star Jan 2, 2021, 7:48 AM

#

why not run?

📎 Ekran_Goruntusu_15.png

lapis sequoia Jan 2, 2021, 10:27 AM

#

!code

arctic wedgeBOT Jan 2, 2021, 10:27 AM

#

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

hybrid scroll Jan 2, 2021, 11:04 AM

#

Hi guys, is it possible to get probability of classes in clustering with k=2?

fleet heath Jan 2, 2021, 11:21 AM

#

hybrid scroll Hi guys, is it possible to get probability of classes in clustering with k=2?

That is not a wise thing to do, you should always choose odd value for k

hybrid scroll Jan 2, 2021, 11:25 AM

#

fleet heath That is not a wise thing to do, you should always choose odd value for `k`

but I already know the label only have binary class, so the k is fixed to 2

obsidian delta Jan 2, 2021, 11:29 AM

#

Hello, I have used Elastic Net model to use several columns of my dataframe to create a prediction column. Does anyone know how to now make it so I can give a new value input value for one of the columns and have it calculate a new prediction? (given new X, predict y)

fleet heath Jan 2, 2021, 11:30 AM

#

hybrid scroll but I already know the label only have binary class, so the k is fixed to 2

k is not the number of classes, it is the number of neighbours that will be checked for a given data point

hybrid scroll Jan 2, 2021, 11:31 AM

#

fleet heath `k` is not the number of classes, it is the number of neighbours that will be ch...

sorry my bad, I mean the cluster is 2

#

I dont have experience in unsupervised learning, sometimes I still confused tho

#

So, is it possible to get probability in each class with number of cluser = 2?

fleet heath Jan 2, 2021, 11:34 AM

#

Yes, you can get probability of a new data point if you have 2 clusters

hybrid scroll Jan 2, 2021, 11:35 AM

#

fleet heath Yes, you can get probability of a new data point if you have 2 clusters

Do you mind to share the sklearn tutorial for it?

fleet heath Jan 2, 2021, 11:36 AM

#

hybrid scroll Do you mind to share the sklearn tutorial for it?

https://www.datacamp.com/community/tutorials/k-nearest-neighbor-classification-scikit-learn

DataCamp Community

KNN Classification using Scikit-learn

Learn K-Nearest Neighbor(KNN) Classification and build KNN classifier using Python Scikit-learn package.

#

Here the two classes are yes and no

obsidian delta Jan 2, 2021, 11:40 AM

#

obsidian delta Hello, I have used Elastic Net model to use several columns of my dataframe to c...

Can anyone point me in the right direction? I've tried looking how to do this but am a bit stuck on this

lapis sequoia Jan 2, 2021, 3:12 PM

#

do u know any AI to remove background that can be used on python pls?
If so, could u give me the link to download?

sullen crescent Jan 2, 2021, 3:29 PM

#

lapis sequoia do u know any AI to remove background that can be used on python pls? If so, cou...

you can use OpenCV instead

#

grabcut or watershed method

lapis sequoia Jan 2, 2021, 3:37 PM

#

how opencv?

#

the background is not uniform

subtle grotto Jan 2, 2021, 4:00 PM

#

Hey, i just started going thru Dataquest to learn python, and im trying to put into mathematical sense/logical sentences, what im learning. let me give you an example:

"for row in apps_data:"
for every row in apps_data, do this

im having trouble deciphering how this part works...
N=11

for value in range(N+1):
print(value)

how does this create a list of: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12?

if N already is assigned 11, how does 'value' not just equal 12?

gray ibex Jan 2, 2021, 4:10 PM

#

range(N+1) mean you loop from 0 to N

subtle grotto Jan 2, 2021, 4:16 PM

#

So specifically in a for loop, 'range' will print starting from where I want?

gray ibex Jan 2, 2021, 4:18 PM

#

no your code is right

#

for i in range(N+1):
print(i)

#

it will print 0 , 1, 2, .. 11

#

if you assign N=11

lapis sequoia Jan 2, 2021, 4:19 PM

#

guys i have a problem

#

https://colab.research.google.com/github/shreyas-bk/U-2-Net-Demo/blob/master/DEMOS/U_2_Netp_Demonstration_Colab.ipynb#scrollTo=HSt4wakMS-wc

Google Colaboratory

#

I am not understanding this

#

I mean, I though U-2-Net was a NN

#

But the code here is using full opencv, i dont see the neural network anywhere

subtle grotto Jan 2, 2021, 4:20 PM

#

gray ibex if you assign N=11

Thanks!

gray ibex Jan 2, 2021, 4:21 PM

#

the source is https://github.com/shreyas-bk/U-2-Net right ?

GitHub

shreyas-bk/U-2-Net

Modified fork of NathanUA's U-2-Net Repository. Used for demonstration purposes. - shreyas-bk/U-2-Net

lapis sequoia Jan 2, 2021, 4:21 PM

#

yes

#

i mean, there is a model and stuff

#

but i dont see the model being used at all

gray ibex Jan 2, 2021, 4:22 PM

#

in your colab, the said run u2net_test.py

#

*they

lapis sequoia Jan 2, 2021, 4:23 PM

#

there is a cell called

#

Results

#

if u inspect the code

#

is pure opencv and numpy

#

there is no model there

#

but after cloning it repository, there is a folder called model

#

so i am a bit lost

gray ibex Jan 2, 2021, 4:24 PM

#

i think they hide the code

lapis sequoia Jan 2, 2021, 4:24 PM

#

u can see it

#

double click the cell

#

code opens

gray ibex Jan 2, 2021, 4:26 PM

#

line 5

#

they call the .py file

lapis sequoia Jan 2, 2021, 4:27 PM

#

ah okey

#

so what is the rest?

#

like

#

u can comment that line

#

and the code still working

gray ibex Jan 2, 2021, 4:29 PM

#

when you run .py file it's will create the output but not display it

#

the cv2 below does the display

#

display in colab

lapis sequoia Jan 2, 2021, 4:32 PM

#

rlly?

#

i think the cv2 below does the remove background thing

#

ah

#

true

#

so everything below

#

is to display what?

#

the bounding box and the mask coloured?

gray ibex Jan 2, 2021, 4:35 PM

#

highlight them

lapis sequoia Jan 2, 2021, 4:35 PM

#

okey

gray ibex Jan 2, 2021, 4:35 PM

#

in line 23 they load output image from saved dir

lapis sequoia Jan 2, 2021, 4:35 PM

#

where are we talking about?

#

the result cell

#

or the .py?

gray ibex Jan 2, 2021, 4:35 PM

#

yeah the result cell

lapis sequoia Jan 2, 2021, 4:36 PM

#

anyway to see line numbers?

#

okey got it

#

okey, that load_image is from keras.preprocessing

#

yeah i was like wtf

#

first lines are opening an image from results

#

and results is empty by default

#

now i see results gets files after calling the model

#

okey so, how can i download the model to my local machine?

#

the model already trained

gray ibex Jan 2, 2021, 4:41 PM

#

the weight is already in the repo

#

so clone the github repo

light warren Jan 2, 2021, 4:42 PM

#

hello, can someone help me with a python problem. how do i add a heading to a data frame ? Also there is meant to be 54 columns in the dataset but when imported into python, it only shows as 1 column. https://gyazo.com/4f44a9a4981a4f67c916502b000e8b17

Gyazo

lapis sequoia Jan 2, 2021, 4:42 PM

#

i mean, i will only want the model and weights. I dont like the way it loads images and stuff

#

the data_loader.py

#

cuz the train.py is to train the model, and i wont

#

and the .pth idk what is it

gray ibex Jan 2, 2021, 4:44 PM

#

the .pth is the weight of pytorch

#

you need it to run the trained model

lapis sequoia Jan 2, 2021, 4:44 PM

#

uuuh

#

i dont have pytorch

#

do i need it?

gray ibex Jan 2, 2021, 4:44 PM

#

pip install pytorch

#

yeah

lapis sequoia Jan 2, 2021, 4:44 PM

#

i have keras and tf

gray ibex Jan 2, 2021, 4:44 PM

#

if you want to use this repo, you must have pytorch

obsidian delta Jan 2, 2021, 4:45 PM

#

I calculated RMSE and R2 for predictions, and compared them to the column that corresponds most to the target column.

My predicted column has RMSE of 25,74 and a R2 of 0.65 (seems low)
The column A had RMSE of 31,31 and R2 of 0.69

Does anyone know how it is possible that predicted column has lower RMSE but also lower R2?

Shouldn't lower RMSE equal higher R2?

lapis sequoia Jan 2, 2021, 4:45 PM

#

isnt pytorch just a framework? like, a model is a model

#

no matter if made with pytorch or keras or what ever

#

right?

gray ibex Jan 2, 2021, 4:46 PM

#

no

#

it trained by pytorch

#

you need pytorch to load .pth

lapis sequoia Jan 2, 2021, 4:46 PM

#

yeah, but weights are just numbers for each neuron

gray ibex Jan 2, 2021, 4:46 PM

#

yeah

#

but the way you define the model

#

you need a framewrok

#

the above repo use pytorch to do this

lapis sequoia Jan 2, 2021, 4:47 PM

#

yeah, i understand that

#

but once u do model.save

#

shouldnt the model be just a model?

#

pytorch is based on tf

#

and keras too

sullen crescent Jan 2, 2021, 4:49 PM

#

no, pytorch based on torch developed by facebook

lapis sequoia Jan 2, 2021, 4:49 PM

#

i dont care to be honest. My point is that a model is a model, no matter how u create it

gray ibex Jan 2, 2021, 4:49 PM

#

@sullen crescent is right

lapis sequoia Jan 2, 2021, 4:50 PM

#

even if u create a model of 1 layer with 8 neuros, it will be a model with 1 layer and 8 neurons if u use keras, or pytorch, or tensorflow, or what ever

sullen crescent Jan 2, 2021, 4:50 PM

#

u cant load model based on framework A to be loaded on framework B without converting it first

lapis sequoia Jan 2, 2021, 4:50 PM

#

uuum

#

can i convert this repo model to keras one?

#

i would like having this one on keras

gray ibex Jan 2, 2021, 4:53 PM

#

you can search u2net keras yo use another repo

lapis sequoia Jan 2, 2021, 4:53 PM

#

since i use keras always, not pytorch

#

if possible

gray ibex Jan 2, 2021, 4:53 PM

#

*to

lapis sequoia Jan 2, 2021, 4:53 PM

#

https://github.com/ypeleg/U-2-Net-Keras

GitHub

ypeleg/U-2-Net-Keras

Keras port of U-2-Net. Contribute to ypeleg/U-2-Net-Keras development by creating an account on GitHub.

#

???

#

XD

#

okey i think i got it

#

now... i am windows user

#

i think i dont have git clone

#

i have to manually download?

sullen crescent Jan 2, 2021, 4:55 PM

#

u could convert pytorch model to tensorflow using onnx-tf

lapis sequoia Jan 2, 2021, 4:55 PM

#

will keras load it? cuz i am using keras

#

not tf.keras

gray ibex Jan 2, 2021, 4:55 PM

#

highly recommend swithching to ubuntu

lapis sequoia Jan 2, 2021, 4:55 PM

#

(idk tbh)

lapis sequoia Jan 2, 2021, 4:56 PM

#

gray ibex highly recommend swithching to ubuntu

ubuntu isnt that friendly uwu

sullen crescent Jan 2, 2021, 4:56 PM

#

In Tensorflow 2.0 tf.keras will be the default and I highly recommend to start working using tf.keras

light warren Jan 2, 2021, 4:56 PM

#

hello, can someone help me with a python problem. how do i add a heading to a data frame ? Also there is meant to be 54 columns in the dataset but when imported into python, it only shows as 1 column. https://gyazo.com/4f44a9a4981a4f67c916502b000e8b17

Gyazo

lapis sequoia Jan 2, 2021, 4:57 PM

#

sullen crescent In Tensorflow 2.0 tf.keras will be the default and I highly recommend to start w...

okey okey

#

so i can uninstall keras?

sullen crescent Jan 2, 2021, 4:58 PM

#

it depends on you, if you're still using keras as a standalone api or choose to integrate it with tensorflow v2

lapis sequoia Jan 2, 2021, 4:58 PM

#

uuuh

#

i think i have tf 2.0

#

tensorflow 2.3.1

#

Keras 2.2.4
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.2

sullen crescent Jan 2, 2021, 4:59 PM

#

since you found the implementation of U net for keras, its not an urgent to upgrade to tf.keras

lapis sequoia Jan 2, 2021, 4:59 PM

#

well

#

i found it

sullen crescent Jan 2, 2021, 4:59 PM

#

lapis sequoia uuuh

use tf.keras from now on then

lapis sequoia Jan 2, 2021, 5:00 PM

#

the link i posted above

#

has this

#


Python 3.6
numpy 1.15.2
scikit-image 0.14.0
python-opencv PIL 5.2.0
PyTorch 0.4.0
torchvision 0.2.1
glob```

#

do i still need pytorch???

sullen crescent Jan 2, 2021, 5:03 PM

#

if you dont have pytorch in your pip -list then you have to install pytorch it

lapis sequoia Jan 2, 2021, 5:03 PM

#

okey q.q

#

then why it is called u2net keras

#

if pytorch is needed too? XD

#

anyway, how do i use the model?

#

Merry Christmas

#

like, i used to

model = keras.models.load_model()
model.predict()```

gray ibex Jan 2, 2021, 5:04 PM

#

lapis sequoia Merry Christmas

XDDD

sullen crescent Jan 2, 2021, 5:05 PM

#

lapis sequoia https://github.com/ypeleg/U-2-Net-Keras

this one doesnt use pytorch

lapis sequoia Jan 2, 2021, 5:05 PM

#

sullen crescent this one doesnt use pytorch

if u go to required libraries there is pytorch

sullen crescent Jan 2, 2021, 5:06 PM

#

?

📎 unknown.png

lapis sequoia Jan 2, 2021, 5:07 PM

#

?

#

?

#

❔

#

❓

#

oh

#

i was on the other one 😄

#

sorry hahaa

#

okey, i cloned the repo, and downloaded both models

#

u2net and u2netp (the p one is smaller)

#

now... If i have a proyect on pycharm, how can i use the model from there?

#

like model.load(); model.run(img)??

#

Can someone help me, I am trying to figure out how to write a bot that will read a bunch of text and than replicate it, I already have the data but have no idea where to start on the bot

#

#discord-bots

#

pardon

#

@sullen crescent

#

@gray ibex maybe u are here? u.u

#

They’re busy having conversations with me, sir.

#

:/

#

Haha

south wedge Jan 2, 2021, 5:46 PM

#

howdy pythonizers ?

#

howdy survivings ?

#

hello anyone alive ?

lapis sequoia Jan 2, 2021, 6:03 PM

#

pepedewgong

#

can someone tell me how to download the model from here https://github.com/ypeleg/U-2-Net-Keras and use it on my local pycharm proyect please?

GitHub

ypeleg/U-2-Net-Keras

Keras port of U-2-Net. Contribute to ypeleg/U-2-Net-Keras development by creating an account on GitHub.

#

I have created a reddit bot that gave me a bunch of submissions, I am looking to write a program that reads all of them, and than writes its own submission, a sort of general summary of everything its read. I have seen similar programs do so with movies and stories. I am not looking for help with the specific code, just a general plan of how to do this

#

@velvet thorn can u help me please?

copper kindle Jan 2, 2021, 7:17 PM

#

I got a question regarding hypothesis testing. anybody up for helping a lad out ?

lapis sequoia Jan 2, 2021, 7:19 PM

#

I want to create a new python environment and I typed conda create -n py27 python=2.7 anaconda. Is it normal that it downloads lots of stuff?

#

📎 unknown.png

#

could use some help wih CNNs and ANNS

lapis sequoia Jan 2, 2021, 7:22 PM

#

lapis sequoia could use some help wih CNNs and ANNS

??? Me?

#

Could I send you the error im getting?

lapis sequoia Jan 2, 2021, 7:23 PM

#

lapis sequoia Could I send you the error im getting?

I am new to this, I don't think I can help you

#

lemon_pensive

#

Have anybody tried to prove the existence of God using data analysis?

#

lemon_thinking

serene scaffold Jan 2, 2021, 7:32 PM

#

lapis sequoia Have anybody tried to prove the existence of God using data analysis?

You'd have to decide what proves or disproves that a god exists first

serene scaffold Jan 2, 2021, 7:34 PM

#

lapis sequoia I want to create a new python environment and I typed conda create -n py27 pytho...

Why are you wanting to use 2.7?

lapis sequoia Jan 2, 2021, 7:35 PM

#

serene scaffold Why are you wanting to use 2.7?

I am trying to learn machine learning in audicity.com and they use python 2.7

serene scaffold Jan 2, 2021, 7:35 PM

#

lapis sequoia I am trying to learn machine learning in audicity.com and they use python 2.7

I wouldn't use that website, then

#

Python 2 is deprecated, so it's likely that that website is old. Check the pins for this channel for some other resources

lapis sequoia Jan 2, 2021, 7:38 PM

#

serene scaffold Python 2 is deprecated, so it's likely that that website is old. Check the pins ...

Ok thank you

#

I tried to learn with the coursera course by Andrew Ng but that one is kind of old

quick igloo Jan 2, 2021, 7:46 PM

#

Where did you know data science

lapis sequoia Jan 2, 2021, 7:48 PM

#

quick igloo Where did you know data science

Where did I know?

quick igloo Jan 2, 2021, 7:49 PM

#

lapis sequoia Where did I know?

Where did you go toxlearn

lapis sequoia Jan 2, 2021, 7:51 PM

#

I am currently reading "Python for Data Analysis"

#

I am a math student but I am interest in data science

#

I start the coursera course and the udacity course on machine learning but the udacity course uses outdated programs

#

and the coursera one does not use python

lapis sequoia Jan 2, 2021, 7:54 PM

#

quick igloo Where did you go toxlearn

I am right now on the edx page recommended in the pin of this channel

quick igloo Jan 2, 2021, 7:57 PM

#

lapis sequoia I am right now on the edx page recommended in the pin of this channel

What s your proposal

lapis sequoia Jan 2, 2021, 7:58 PM

#

I want to prove that God is real using data

limpid oak Jan 2, 2021, 8:53 PM

#

I have this code

#

MUSTIL_NO_file_copy[['HB_NO','VILLAGE_NAME', 'DISTRICT_NAME','TEHSIL_NAME']] = [HB_No_extract(filename),VillageName_extract(filename), DISTRICT_NAME,TEHSIL_NAME]

#

but in console it showing this msg

#

usr/local/lib/python3.6/dist-packages/geopandas/geodataframe.py:853: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

#

I don't understood what it means and solution

#

I tried MUSTIL_NO_file_copy.loc[:,'HB_NO','VILLAGE_NAME', 'DISTRICT_NAME','TEHSIL_NAME'] = [HB_No_extract(filename),VillageName_extract(filename), DISTRICT_NAME,TEHSIL_NAME]

#

but showing slicing error

#

please reply

lapis sequoia Jan 2, 2021, 9:39 PM

#

I found a way to do the Udacity course on machine learning using python 3, just download this folder https://github.com/flyinactor91/ud120-projects instead of the one they give you in the course.

GitHub

flyinactor91/ud120-projects

Starter project code for students taking Udacity ud421 - flyinactor91/ud120-projects

lapis sequoia Jan 2, 2021, 10:37 PM

#

I am looking for help extracting nested content of a json file into a pandas df;

{
 "result_type": "EXTENDED",
 "success": true,
 "projects": [
   {
     "project_id": 92770,
     "created_at": 1608319219000,
     "updated_at": 1608319219000
   },
   {
     "project_id": 91407,
     "created_at": 1606510177000,
     "updated_at": 1606510177000
   }
 ]
}```

#

I this exemple I want to import the projects only, and ignore the first 2 objects

#

any help or suggestions would be appreciated, I've been trying out a few solutions (i.e. json normalize) but nothing has worked so far

trim oar Jan 3, 2021, 12:15 AM

#

lapis sequoia any help or suggestions would be appreciated, I've been trying out a few solutio...

You first convert the json object to a dictionary through hello = json.loads(object), then you normalize through sliceing pd.json_normalize(hello['projects'])

late jackal Jan 3, 2021, 1:10 AM

#

I know this isn't strictly python related but if I wanted to include SQL on an engineering resume where should I look for SQL certifications/courses ?

trim oar Jan 3, 2021, 1:42 AM

#

late jackal I know this isn't strictly python related but if I wanted to include SQL on an e...

#career-advice is a better place

late jackal Jan 3, 2021, 1:43 AM

#

ah ty

#

ill copy it over there in a bit thanks

torpid cave Jan 3, 2021, 4:04 AM

#

@limpid oak it relies on how Python works

#

Try using .loc

#

when assigning values with slicers

lapis sequoia Jan 3, 2021, 6:32 AM

#

lapis sequoia I am looking for help extracting nested content of a json file into a pandas df;...

You could extract the values of each, project id,created at and updated at from these keys

#

Into a list

#

Df = pd.Dataframe(zip(list1,list2,list3),columns=[‘project_id’,’created_at’,’updated_at])

#

This should work ig

#

The lists should contain the values which you retrieve

#

list1 = [i[‘project_id’] for i in the_json_file]

#

Similarly for created at and updated at

#

Ping me if u have further queries

lapis sequoia Jan 3, 2021, 11:23 AM

#

Hiiii

#

I am a beginner

#

I am using Ipython, I zoom the screen to write code

#

but there is something that bothers me

#

when I got to the bottom of the screen

#

my code just hide down and then I have to scroll down to see it

#

I usually use control L

#

and my code goes to the top of the screen

#

but

#

I want to be able to write code such that the screen follows me when I got to the bottom of it

#

Is it there something I am missing

#

Or I have to live with this problem?

#

I want to type like this but when I pressed enter

📎 unknown.png

#

When I press enter the screen do not show me the last line of code

📎 unknown.png

#

lemon_grumpy

lapis sequoia Jan 3, 2021, 12:35 PM

#

@lapis sequoia hey help me please with this

lapis sequoia Jan 3, 2021, 12:35 PM

#

lapis sequoia When I press enter the screen do not show me the last line of code

This

#

i didnt get you could you explain again😅

#

you want the whole dataframe to appear?

#

is that what you are saying?

#

or...

#

@lapis sequoia

lapis sequoia Jan 3, 2021, 12:40 PM

#

lapis sequoia <@456226577798135808>

Let's say I am on line [12] and I write some code on this line.

#

yeah

#

I mean line 12

lapis sequoia Jan 3, 2021, 12:40 PM

#

lapis sequoia I want to type like this but when I pressed enter

Like this

#

Then I press enter

#

if you press enter that command ends

lapis sequoia Jan 3, 2021, 12:41 PM

#

lapis sequoia if you press enter that command ends

🤔

#

I am very very new

#

To this

#

Let me explain futher

#

Wait

#

k

#

So I press enter

#

yeah

#

go on

lapis sequoia Jan 3, 2021, 12:42 PM

#

lapis sequoia When I press enter the screen do not show me the last line of code

And the screen turn to this

#

Where it shows me

#

Line 11

#

But I was in line 12

#

Then

#

I have to scroll down

#

To see the code I write in line 12

#

ohhh ok

#

got it

#

Tupega

#

idt there is a solution to this and personally i dont work on command line much for data science

#

use jupyter

#

or colab

#

hosted notebooks are so g

lapis sequoia Jan 3, 2021, 12:44 PM

#

lapis sequoia idt there is a solution to this and personally i dont work on command line much ...

Oh. I understand

#

I am just working on command line because of the book I am reading

#

im sure u can implement those same things in jupyter too

#

i mean i for one do

lapis sequoia Jan 3, 2021, 12:45 PM

#

lapis sequoia im sure u can implement those same things in jupyter too

I am going to try it

#

Thanks

lapis sequoia Jan 3, 2021, 12:46 PM

#

lapis sequoia I am going to try it

sure

lapis sequoia Jan 3, 2021, 12:46 PM

#

lapis sequoia Thanks

anytime

mint palm Jan 3, 2021, 1:39 PM

#

i think your issue is dependent on what you are using and nothing to do with the code you are writing

#

@lapis sequoia

lapis sequoia Jan 3, 2021, 1:41 PM

#

mint palm i think your issue is dependent on what you are using and nothing to do with the...

I am using Ipython so you are probably right

#

I am going to use Jupyter from now on

#

MIT Machine Learning Professor plays The Stanley Parable I think this is fun to watch. https://youtu.be/OIEIr4wtVvU

YouTube

Lex Fridman

Lex plays The Stanley Parable

Simulated existential crisis in the form of a video game. Please check out our sponsors:

Tryolabs: https://tryolabs.com/lex
Vincero: https://vincerowatches.com/lex to get up to 25% off + free shipping

OUTLINE:
0:00 - Introduction
0:48 - Round 1 - The Matrix
8:12 - Round 2 - Reincarnation
9:58 - Round 3 - Winning
12:39 - Round 4 - Adventure ...

▶ Play video

#

😂 🤣

mint palm Jan 3, 2021, 1:53 PM

#

i am doing neural network for fun..........lol my specialization is in mechanical engg

#

just cuz i started to like coding

lapis sequoia Jan 3, 2021, 2:02 PM

#

mint palm i am doing neural network for fun..........lol my specialization is in mechanica...

I am a mathematician and I am learning this for fun as well😀

mint palm Jan 3, 2021, 2:03 PM

#

gr8

#

infact i want career in software engg lol

#

😅

lapis sequoia Jan 3, 2021, 2:33 PM

#

mint palm infact i want career in software engg lol

That is cool

#

@mint palm What learning resource are you using?

mint palm Jan 3, 2021, 2:42 PM

#

coursera edx so far

#

but i have to agree they are 20 percent

#

80 percent is curiosity and stacksoverflow

#

@lapis sequoia

#

you guys offcourse

lapis sequoia Jan 3, 2021, 2:44 PM

#

@mint palm Cool I am also using those plus Udacity (this one has some criticism by some people but I think is ok)

#

Also I am reading books

mint palm Jan 3, 2021, 2:45 PM

#

yeah thats a nice site too

lapis sequoia Jan 3, 2021, 2:45 PM

#

I found the edx course to be a little heavy for a beginner like me tho

mint palm Jan 3, 2021, 2:46 PM

#

i feel coursera has a really good neural network teacher

#

hes the founder of coursera himself

#

he teaches great

#

andrew ng

lapis sequoia Jan 3, 2021, 2:46 PM

#

mint palm i feel coursera has a really good neural network teacher

I will try Neural Networks later, I am focus on Machine Learning right now

lapis sequoia Jan 3, 2021, 2:47 PM

#

mint palm andrew ng

Yeah I heard these are good courses

mint palm Jan 3, 2021, 2:47 PM

#

neural network is inside machine learning

#

its like a tool for machine ;earning

#

u will eventually come across it in machine learning at some point

lapis sequoia Jan 3, 2021, 2:49 PM

#

mint palm its like a tool for machine ;earning

I see, is just that there are different courses and books with the title "Neural Network" so I though that it was a different branch of AI

mint palm Jan 3, 2021, 2:50 PM

#

oh yeah correct

lapis sequoia Jan 3, 2021, 2:50 PM

#

@mint palm Do you think I need to learn machine learning first then Neural Network?

#

Or it does not matter the order

lapis sequoia Jan 3, 2021, 3:38 PM

#

I am reading a book called "Python for Data Analysis" by the creator of pandas but I am thinking "Is not the same to just read the pandas and numpy documentation?"

#

Plus the book do not have exercises in which on can practice the things

pseudo moss Jan 3, 2021, 3:39 PM

#

I am quite new to as well.....but i can tell for sure that if you can solve some intermediate algorithm problems in the language and know how the language works then you will not have any problem in understanding NEURAL NETWORKS

lapis sequoia Jan 3, 2021, 3:39 PM

#

pseudo moss I am quite new to as well.....but i can tell for sure that if you can solve some...

O.k THANK YOU.

pseudo moss Jan 3, 2021, 3:40 PM

#

The course i am taking on coursera covers the numby part that is required to implement a neutal network

#

Np

#

I can share you the course link if you want

lapis sequoia Jan 3, 2021, 3:41 PM

#

pseudo moss I can share you the course link if you want

Yes please

#

which library is better for plotting

north plinth Jan 3, 2021, 4:10 PM

#

I guess matplotlib

light warren Jan 3, 2021, 4:27 PM

#

hello, can someone help me with a python problem. how do i add a heading to a data frame ? Also there is meant to be 54 columns in the dataset but when imported into python, it only shows as 1 column. https://gyazo.com/4f44a9a4981a4f67c916502b000e8b17

Gyazo

proven sigil Jan 3, 2021, 4:30 PM

#

light warren hello, can someone help me with a python problem. how do i add a heading to a da...

Looks like the data is not comma separated. You need to use the correct delimiter (tab from the looks of it) when loading the file.

light warren Jan 3, 2021, 4:34 PM

#

do you know i can change the delimiter

proven sigil Jan 3, 2021, 4:34 PM

#

How are you loading the file

light warren Jan 3, 2021, 4:35 PM

#

https://gyazo.com/e27a743890e2d8fb0f14e48db465fc9b

Gyazo

#

df4= pd.read_csv(data4,header=None)

proven sigil Jan 3, 2021, 4:37 PM

#

df4= pd.read_csv(data4,header=None,sep='\t')

light warren Jan 3, 2021, 4:38 PM

#

still comes up with 1 column

proven sigil Jan 3, 2021, 4:38 PM

#

df4 = pd.read_csv(data4, header=None, sep=' ')

light warren Jan 3, 2021, 4:38 PM

#

it works now

#

thanks soo much

proven sigil Jan 3, 2021, 4:38 PM

#

Np!

#

'\s+'

#

This is better

light warren Jan 3, 2021, 4:39 PM

#

another question, do u know how i can add header to the data?

light warren Jan 3, 2021, 4:40 PM

#

proven sigil '\s+'

so df4 = pd.read_csv(data4, header=None, sep='\s+')?

proven sigil Jan 3, 2021, 4:40 PM

#

Yes

proven sigil Jan 3, 2021, 4:41 PM

#

light warren another question, do u know how i can add header to the data?

rename_columns or df4.columns = [list_of_columns]

light warren Jan 3, 2021, 4:44 PM

#

thanks sooo much again

proven sigil Jan 3, 2021, 5:08 PM

#

:)

lapis sequoia Jan 3, 2021, 5:09 PM

#

Need

#

Need Help with Parsing server llogs from laravel :
Hello need help with parsing it and converting it to csv.
I have Thought about two nested loops . One reading each line of the text line and other parses and writes until 5 and starts again .
Logfile:
`location: /user/create
[ERROR]: Cant Connect to database
[ip]: 127.0.0.1
[line]:/middleware/resources(112)

[ERROR]: Cant Connect to database
[ip]: 127.0.0.1
[line]:/middleware/resources(112)
-------------------------------------`

loud spoke Jan 3, 2021, 5:20 PM

#

is this related to data science?

#

(no)

#

i am maybe wrong

hard hound Jan 3, 2021, 5:20 PM

#

Hey does anyone here work wth google colab?

loud spoke Jan 3, 2021, 5:21 PM

#

Yep

#

i do

#

@hard hound

hard hound Jan 3, 2021, 5:21 PM

#

is it good?

loud spoke Jan 3, 2021, 5:21 PM

#

It is

#

For projects

#

i use it

hard hound Jan 3, 2021, 5:22 PM

#

is it to replace jupyter notebooks

loud spoke Jan 3, 2021, 5:22 PM

#

It IS a jupyter notebook

#

lol

hard hound Jan 3, 2021, 5:22 PM

#

could you explain it to me if you are available?

loud spoke Jan 3, 2021, 5:23 PM

#

No, im not available rn

hard hound Jan 3, 2021, 5:23 PM

#

well okay

#

Hey could someone help me

#

I am getting error while cleaning data

#

ValueError: A given column is not a column of the dataframe

loud spoke Jan 3, 2021, 5:24 PM

#

¯_(ツ)_/¯

hard hound Jan 3, 2021, 5:24 PM

#

i have checked everything

hard hound Jan 3, 2021, 5:24 PM

#

loud spoke ¯\_(ツ)_/¯

hey dont think I am a noob

#

Help

📎 Screenshot_2021-01-03_225617.png

serene scaffold Jan 3, 2021, 5:47 PM

#

hard hound Help

I personally don't recommend notebooks for beginners because they can be confusing to debug if you aren't familiar with how cells of code work.

There's not enough information here to know what went wrong. Could you copy and paste the code as text, and copy and paste the error message as text? The error message is cut off in your screenshot.

hard hound Jan 3, 2021, 5:49 PM

#

serene scaffold I personally don't recommend notebooks for beginners because they can be confusi...

Hey no offence but I am not a beginner

hard hound Jan 3, 2021, 5:49 PM

#

serene scaffold I personally don't recommend notebooks for beginners because they can be confusi...

but thanks for reply

serene scaffold Jan 3, 2021, 5:49 PM

#

hard hound Hey no offence but I am not a beginner

sorry; I misread your message when you said "hey dont think I am a noob". I thought you were saying that you think of yourself as a noob. My apologies.

#

(I don't find the word "noob" helpful, personally. I just thought you were calling yourself that.)

hard hound Jan 3, 2021, 5:51 PM

#

hey seem cool

serene scaffold Jan 3, 2021, 5:51 PM

#

!code

arctic wedgeBOT Jan 3, 2021, 5:51 PM

#

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

serene scaffold Jan 3, 2021, 5:51 PM

#

^ whenever you're ready, just include your code and error message like that.

hard hound Jan 3, 2021, 5:51 PM

#

📎 Screenshot_2021-01-03_232113.png

#

heres the screenshot of error

#

well i do have another method with pandas but i want to solve the error with sklearn

serene scaffold Jan 3, 2021, 5:53 PM

#

hard hound

Text of text is preferable to screenshots of text. That aside, do you know what dataframe the error message is talking about?

hard hound Jan 3, 2021, 5:53 PM

#

pandas

serene scaffold Jan 3, 2021, 5:54 PM

#

hard hound pandas

Right, Pandas is the library that has the DataFrame class, but which specific dataframe do you think the error is about?

#

what kind of data is it? I guess I would check to see what algorithms have been demonstrated to be predictive for that kind of data.

#

looks like stocks?

#

ah. I guess let's see what models have been predictive for future stock performance.

#

is this more about learning about AI, or is this going to be part of a "real" project?

#

This article looks pretty comprehensive, at least from a learning perspective: https://towardsdatascience.com/using-deep-learning-ai-to-predict-the-stock-market-9399cf15a312

Medium

Using Deep Learning AI to Predict the Stock Market

Forecasting Stock Prices with Neural Networks containing Multivariable Inputs from Technical Analysis

#

I've not heard of torchtext.

hard hound Jan 3, 2021, 6:07 PM

#

hey does anyone use their brokers api?

#

the one you trade with

upbeat storm Jan 3, 2021, 7:41 PM

#

like robinhood

#

hes asking if you use robinhood or something like webull's api

fallow prism Jan 3, 2021, 9:50 PM

#

seed = 2201433561
random.seed(seed)
np.random.seed(seed)
model = Doc2Vec(corpus, vector_size=vector_size, seed=1)
pca = PCA(n_components=2, svd_solver='full', whiten=True,random_state=seed)
HELP! this is not working, i can't make the same vector twice 😭

digital niche Jan 4, 2021, 12:17 AM

#

Wrote a script to automatically collect gas meter readings from video. The service is running on my home k8s cluster and I'm doing a Notebook to go over the functions/algorithms I used. It was a fun but a little frustrating as OpenCV is a little quirky. This is a draft, still need to dive into decoding the analog dials using a heavy dose fudging the numbers.
https://github.com/brontide/Gas-Meter-Notebook/blob/main/gasmeter.ipynb

GitHub

brontide/Gas-Meter-Notebook

Colab Notebook for OpenCV gasmeter readins. Contribute to brontide/Gas-Meter-Notebook development by creating an account on GitHub.

livid quartz Jan 4, 2021, 7:44 AM

#

Whats up with this error? It only appears when I fit my ML model to undersampled data, but it does not appear when fit to oversampled or imbalanced data

#

"Precision is ill-defined and being set to 0.0 due to no predicted samples. Use zero_division parameter to control this behavior."

tepid hamlet Jan 4, 2021, 8:01 AM

#

What does "data prediction" even mean? And no, PyTorch Text is not for "text files" it's an API for solving problems involving text data.

lapis sequoia Jan 4, 2021, 10:03 AM

#

can i use multiple csv in plot

hidden halo Jan 4, 2021, 10:45 AM

#

I have a pandas dataframe with a date column. I want to convert that column into a list of dates.
However, if I simply do list(df['date']) it gives me a list iwth numpy.datetime64() datatype. if I do df['date'].tolist() it converts the dates to integers.
How do I convert the dates to datetime.datetime data type?

lapis sequoia Jan 4, 2021, 12:23 PM

#

pd.to_datetime(list(df[‘date’])) ?

lapis sequoia Jan 4, 2021, 12:59 PM

#

can someone help me out with this error please?

#

📎 unknown.png

#

am trying to do a machine learning model

#

whether or not a person wears mask

digital niche Jan 4, 2021, 1:44 PM

#

@lapis sequoia it doesn't understand the file you're giving it.

lapis sequoia Jan 4, 2021, 1:44 PM

#

oh but how do I fix it ?

digital niche Jan 4, 2021, 1:46 PM

#

I've never used tensorflow, are you sure you're calling load_img correctly?

lapis sequoia Jan 4, 2021, 1:47 PM

#

yeah

#

train test folders are in the same directory as the file

odd yoke Jan 4, 2021, 1:52 PM

#

did you do import PIL.Image as pil_image ? @lapis sequoia

#

just to make sure that this is PIL.Image.open

#

oh wait nvm that's the internals of keras

lapis sequoia Jan 4, 2021, 1:59 PM

#

yeah

quiet patio Jan 4, 2021, 2:04 PM

#

hey does anyone know how to handle an error in lambda function

#

📎 unknown.png

lapis sequoia Jan 4, 2021, 2:06 PM

#

ask in #python-discussion

quiet patio Jan 4, 2021, 2:09 PM

#

thanks

hard hound Jan 4, 2021, 2:22 PM

#

Please help

📎 Screenshot_2021-01-04_195052.png

wild sorrel Jan 4, 2021, 2:57 PM

#

Hey guys, I have the following pandas code

import pandas as pd


def list_of_dicts_to_excel(seller_data, buyer_data, name):
    df_seller = pd.DataFrame.from_dict(seller_data)
    df_seller.set_index("PROPOSAL", inplace=True)
    df_buyer = pd.DataFrame.from_dict(buyer_data)
    df_buyer.set_index("PROPOSAL", inplace=True)
    with pd.ExcelWriter(f"{name}.xlsx", engine="openpyxl", mode="a") as writer:
        df_seller.to_excel(writer, sheet_name="Buyers")
        df_buyer.to_excel(writer, sheet_name="Sellers")

in which I wanted to make 2 sheets, one named "buyers" and the other named "sellers", and then each time I call my list_of_dicts_to_excel function, pandas should write each new data frame in the following available lines, but instead creates a new sheet for each function call. How can I accomplish my goal based on this code?

surreal valley Jan 4, 2021, 3:45 PM

#

Anyone here work with AI? Wanting to do an object detection for different objects in images. My school uses a custom captcha and our teacher is giving us an A if we can make an AI that can solve it through training it with images

#

It’s like googles but doesn’t use the same images

#

Anyone have sources or articles I can read

#

^ tag me if u reply plz

hushed wasp Jan 4, 2021, 6:25 PM

#

Hello,

I need to count in my dataframe the unique values in customer_unique_id and create a new column/variable counting the number of time a customer_unique_id appears, drop rows to only keep one customer_unique_id and eventually create a new category variable.

The Dataframe :

````customer_unique_id       order_id        category
   ef54                     '0024'           gift
   ab58                     '0587'         school
   yg41                     '0678'           gift
   af48                     '0469'           gift
   ef54                     '8514'         school
   af48                     '2771'           gift    ````

The expected Dataframe output :

      ````   customer_unique_id       order_id        category    number_of_orders      category_2
       ef54                     '0024'           gift            2                 school
       ab58                     '0587'         school            1                 Nan
       yg41                     '0678'           gift            1                 Nan
       af48                     '0469'           gift            2                 gift    ````

The most important for me is to understand how to create the variable numer_of_orders, but category_2 would be a bonus.

I don't have more than two orders per customer_unique_id.

Thanks!!

tall trail Jan 4, 2021, 6:41 PM

#

Hey! so i have this weird thing goin on in plotly and its giving me headaches by now lol. does anyone have a clue whats going on with the y axis?

#

📎 unknown.png

#

this is the code for the graph, i can also show you the dataframes

📎 unknown.png

lunar arrow Jan 4, 2021, 8:38 PM

#

Not really a data science but a numpy question, I have a 128 chars per line by 128 lines text file that contains hex data ([0-9,A-F]), and I'm currently reading it with
data = np.genfromtxt("data.txt", dtype=np.uint8, delimiter=1)
It works but my problem is, it reads each character as an entry, I want it to read each half character as an entry (like read each 4 bits, and treat them like an unsigned number between 0 and 16). Also I think it doesn't read the hex characters correctly

mellow vapor Jan 4, 2021, 8:40 PM

#

Suppose i have 11 features and based on their mean value i calculate the MSE,now if user is allowed to modify 2 of the features by his own input,do i need to recalculate the mean squared error and root mean squared error?

fiery cobalt Jan 4, 2021, 10:55 PM

#

i'm having a issue with matplotlib
my xticks aren't showing the way i wanted them to
https://cdn.discordapp.com/attachments/675433470746558465/795787104252592128/image.png
my code ```py
xticks = [i for i in range(len(pyplot.xticks()[0]) + 1) if i % 56 == 0]
pyplot.xticks(xticks, rotation=90)

velvet thorn Jan 4, 2021, 11:02 PM

#

🥴

velvet thorn Jan 4, 2021, 11:03 PM

#

fiery cobalt i'm having a issue with matplotlib my xticks aren't showing the way i wanted the...

what do you expect

#

look into MultipleLocator

#

if you want ticks at multiples of 56

fiery cobalt Jan 4, 2021, 11:04 PM

#

never even knew about that, thanks

velvet thorn Jan 4, 2021, 11:05 PM

#

mellow vapor Suppose i have 11 features and based on their mean value i calculate the MSE,now...

...yes? because now your predictions are different

lime jewel Jan 5, 2021, 5:48 AM

#

Looking for data science help with sklearn's LogisticRegressionCV

Is there anyway to measure the variance in predictions for each fold during training with this method?

Please @ me if you respond to this question!

mellow vapor Jan 5, 2021, 7:30 AM

#

@velvet thorn The course i m studying didn't re-evaluate those values

#

m displaying my code,so that you can get some insight and point me out where i m going wrong

#

# %%
# Loading required Modules
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# %%
# Loading the complete Dataset
boston_dataset = load_boston()
# Building and arranging Data
data = pd.DataFrame(data=boston_dataset.data,
                    columns=boston_dataset.feature_names)
# Optimized Version of the data used in analysis
features = data.drop(['INDUS', 'AGE'], axis=1)
# Taking log of the target values to reduce the skewness
log_prices = np.log(boston_dataset.target)
# Bulding and arranging the Price Dataset
target = pd.DataFrame(log_prices, columns=['PRICE'])
features.head()


# %%
# Create an empty array to replicate the features
property_stats = np.ndarray(shape=(1, 11))
# First row of values will contain the mean values of each feature
property_stats = features.mean().values.reshape(1, 11)

#

# %%
# Performing regression and obtaining mse and rmse
regression = LinearRegression().fit(features, target)
fitted_values = regression.predict(features)
mse = mean_squared_error(target, fitted_values)
rmse = np.sqrt(mse)

# %%
# Perform Prediction based on mean/user inputted values
# Indices based on the Features table
CHAS_IDX = 2
RM_IDX = 4
PTRATIO_IDX = 8


def logEstimate(nr_rooms, ptRatio, river=False, confidence=False):
    # Predict values based on user input if provided
    property_stats[0][RM_IDX] = nr_rooms
    property_stats[0][PTRATIO_IDX] = ptRatio
    property_stats[0][RM_IDX] = 1 if river else 0
    estimate = regression.predict(property_stats)
    if confidence:
        upper_bound = estimate+2*rmse
        lower_bound = estimate-2*rmse
        interval = 95
    else:
        upper_bound = estimate+rmse
        lower_bound = estimate-rmse
        interval = 68
    return estimate, upper_bound, lower_bound, interval

#

def getEstimate(nr_rooms, ptRatio, river=False, confidence=False):
    """
    Estimate a price of Property in Boston.

    Keyword Arguments:

    nr_rooms -- Number of rooms in the property

    PTRATIO -- Number of students per teacher in classroom for nearby schools

    CHAS -- True if property is near the river else 0 

    Confidence -- True if prediction range is 95% accurate or False for 68% accuracy

    """
    # COndition checking
    if(nr_rooms < 1 or nr_rooms > 9):
        print('Invalid Value of Rooms,it cannot exceed 9')
        return
    elif(ptRatio < 1 or ptRatio > 22):
        print("Invalid value for PTRATIO,it cannot exceed 22")
        return
    # By scaling the median we can get the idea of current house Prices
    zillow_current_median = 653.6
    target_medain = np.median(boston_dataset.target)
    scale_factor = zillow_current_median/target_medain
    estimate, upper_bound, lower_bound, interval = logEstimate(
        nr_rooms, ptRatio, river, confidence)
    # Converting log based values to 1000's dollar and scaling them
    estimate = np.around(np.e**estimate*1000*scale_factor, -3)[0][0]
    upper_bound = np.around(np.e**upper_bound*1000*scale_factor, -3)[0][0]
    lower_bound = np.around(np.e**lower_bound*1000*scale_factor, -3)[0][0]
    # Printing the values
    print(f'Estimated Price of the House is : {estimate} $')
    print(
        f'The Range of the Prices can be from {lower_bound} $ to {upper_bound} $')
    print(f'Confidence interval is {interval}%')

#

This getEstimate function is where i predict my values based on user input

#

but MSE is calculated just before that and the instructor in the course didn't calculate the mse again in the function to decide the range

crisp spruce Jan 5, 2021, 10:32 AM

#

So back the common question... 😄 how to get started with ML and data scince? good with programming language and Ongoing is doing math any recommendation on what next?

cyan beacon Jan 5, 2021, 10:41 AM

#

It depends on how you learn best, if you are into practice just give yourself a goal and do it.

#

You can ask around for ideas of beginner projects

crisp spruce Jan 5, 2021, 10:54 AM

#

so its like learn a programming language and maths and jump into small projects apply? nothing additional to learn

cyan beacon Jan 5, 2021, 11:11 AM

#

Years and Years of stuff to learn, but you have to practice otherwise you'll never be able to use anything you've learned

#

At least that's what worked for me.

lapis sequoia Jan 5, 2021, 11:48 AM

#

would someone help me with a map-reduce and hadoop clustering problem?

#

i dont even know where to begin

vital ocean Jan 5, 2021, 11:48 AM

#

lapis sequoia would someone help me with a map-reduce and hadoop clustering problem?

hmm

#

yes?

#

can u just telll?
the prob?

lapis sequoia Jan 5, 2021, 11:51 AM

#

📎 unknown.png

#

whoops

#

sorry

#

I am way too new to this and I dont even know how to begin

vital ocean Jan 5, 2021, 12:08 PM

#

lapis sequoia

Hmm aria You have to scrape the data from NCDC i think

lapis sequoia Jan 5, 2021, 12:13 PM

#

and... how do I do that?

#

Again, wayyy too new at this I have no clue what I am doing

frail socket Jan 5, 2021, 1:09 PM

#

hey data science peps :: can anyone send some reading materials or nice tutorial for dash plotly

#

and what`s in trend for a jazzy illustration library ?

pale hull Jan 5, 2021, 1:11 PM

#

I'll see what I can do

frail socket Jan 5, 2021, 1:17 PM

#

thanks

barren bluff Jan 5, 2021, 2:38 PM

#

Hi Im having issues with my transfer learning project, I just added a question in #help-croissant if anyone is interested. didnt know there was a channel for this in advance

pastel willow Jan 5, 2021, 4:33 PM

#

I'm trying to add an explicit content filter for images using opencv2
does anyone know where I might find a cascade file for detecting NSFW images?

#

~~would rather not have to download thousands of NSFW just to create a cascade file lmao~~

upbeat pagoda Jan 5, 2021, 7:11 PM

#

https://youtu.be/wCOuu0-o5YI

YouTube

StartupBros

What's a "Lightboard" Studio (How Do They Work?)

Here's how we built our Lightboard Video Studio to achieve that cool "Floating Neon Glass Whiteboard" effect — and how to make one yourself with a semi-simple DIY Project 😀

RESOURCES & LINKS:

👉 Master List of Recommended Tools - https://strtpb.ro/resources

Start Here Guide - https://strtpb.ro/start

Free Tools - https://strtpb.ro/free

Priva...

▶ Play video

#

Would it be possible to build a "lightboard" with just a webcam and machine learning

dreamy flume Jan 5, 2021, 7:34 PM

#

hi all, I'm having a bit of trouble using pandas read_excel; is this the right place to ask?

#

I'm trying to read this file directly:
df = pd.read_excel('https://open.alberta.ca/dataset/11245675-b047-49fc-8bd1-cc2ce8314a6d/resource/e8aac308-c754-484c-b446-0c57ed0e8d37/download/baby-names-frequency.xlsx')

But I'm getting this error: xlrd.biffh.XLRDError: Excel xlsx file; not supported

I've tried changing https to http but that hasn't worked either.

#

The docs for read_excel say that the first argument (io) can be a url, so I'm not sure why this doesn't seem to work

glad hound Jan 5, 2021, 7:55 PM

#

hii

ancient lichen Jan 5, 2021, 9:26 PM

#

so I have this code:
https://hastebin.com/roregixuco.py
And I'm running it in google collaboratory, and it's saying that model.fit(training, output, n_epoch=1000, batch_size=8, show_metric=True) is raising IndexError: list index out of range in this inner code:
feed_dict[net_inputs[i]] = x. Idk why, but if anyone knows tensorflow, help would be awesome 🙂

jade zephyr Jan 5, 2021, 10:05 PM

#

Does anyone know how I can access/modify arrays that are inside a dictionary?

velvet thorn Jan 5, 2021, 11:02 PM

#

jade zephyr Does anyone know how I can access/modify arrays that are inside a dictionary?

the same way you would access anything stored in a dict

velvet thorn Jan 5, 2021, 11:02 PM

#

dreamy flume The docs for `read_excel` say that the first argument (`io`) can be a url, so I'...

did you Google this?

#

this was the first result

#

https://stackoverflow.com/questions/65254535/xlrd-biffh-xlrderror-excel-xlsx-file-not-supported

Stack Overflow

xlrd.biffh.XLRDError: Excel xlsx file; not supported

I am trying to read macro enabled excel work sheet using pandas.read_excel with xlrd library, its running fine in local but when I try to push the same into PCF I am getting this error:
2020-12-11T...

manic granite Jan 6, 2021, 12:06 AM

#

I have a big weird problem that is killing me

#

I found a pre-trained model on pytorch using GPU, but i wanna use keras. So i used pytorch2keras library (it uses onnx) to go from pytorch model to keras

#

Now, on colab it works, but i need it to run locally, and i dont have an nvidia GPU

#

So, keras model cries because the image are not on NHWC (batch, heigth, width, channels). My images are with channels first

#

Because the pytorch model was trained like this (i guess on GPU having channels first speeds ups things or w/e)

#

So if i use numpy.transpose, then the model cries because the recieved shape isnt the expected

#

The only fix i have think of is to add an useless layer at the top that recieves the images with the same shape the CPU wants. But since i wanna use the already pre-trained weights, this layer has to do nothing

#

Anyone could help?

wintry nacelle Jan 6, 2021, 12:50 AM

#

someone please help my DCGANs look like shit

📎 dcgan.gif

velvet thorn Jan 6, 2021, 1:03 AM

#

@manic granite what is transform?

#

do you mean transpose?

#

anyway, this is just a question of reshaping

manic granite Jan 6, 2021, 1:09 AM

#

yes, transpose

manic granite Jan 6, 2021, 1:09 AM

#

velvet thorn anyway, this is just a question of reshaping

it is not

#

read my issue carefully again

manic granite Jan 6, 2021, 1:10 AM

#

manic granite So, keras model cries because the image are not on NHWC (batch, heigth, width, c...

^

manic granite Jan 6, 2021, 1:10 AM

#

manic granite So if i use numpy.transpose, then the model cries because the recieved shape isn...

^

velvet thorn Jan 6, 2021, 1:13 AM

#

manic granite read my issue carefully again

I did

#

but

#

okay then

#

if you say so

manic granite Jan 6, 2021, 1:14 AM

#

okey. reshape what?

manic granite Jan 6, 2021, 1:30 AM

#

@velvet thorn

austere swift Jan 6, 2021, 5:12 AM

#

manic granite So, keras model cries because the image are not on NHWC (batch, heigth, width, c...

you can change that in the keras.json in the .keras folder in your user

#

just change "image_data_format": "channels_last" to "image_data_format": "channels_first"

tardy condor Jan 6, 2021, 9:27 AM

#

Hi! May I ask about algorithm for machine learning here？

uncut shadow Jan 6, 2021, 9:32 AM

#

Yeah

tardy condor Jan 6, 2021, 9:33 AM

#

Is there any suggested algorithm if I want to use it for position estimation?

true geode Jan 6, 2021, 11:52 AM

#

Hey, has anyone got any dash/plotly experience here? 🙂

boreal summit Jan 6, 2021, 12:24 PM

#

lime jewel Looking for data science help with sklearn's LogisticRegressionCV Is there anyw...

If you mean cross validation, there's a way to return the prediction for each fold instead of the prediction score. This way, you can measure the variance for each fold prediction.

boreal summit Jan 6, 2021, 12:25 PM

#

lime jewel Looking for data science help with sklearn's LogisticRegressionCV Is there anyw...

Instead of using cross_val_score, use cross_val_predict

manic granite Jan 6, 2021, 1:16 PM

#

austere swift you can change that in the keras.json in the .keras folder in your user

doesnt work

#

Default MaxPoolingOp only supports NHWC on device type CPU

austere swift Jan 6, 2021, 1:52 PM

#

You’re training on cpu???

mortal trout Jan 6, 2021, 1:52 PM

#

can someone help in a pandas related thing

#

i have a git project i just need to predict for custom input

#

it predicts for a tsv file

manic granite Jan 6, 2021, 2:11 PM

#

austere swift You’re training on cpu???

no, the model was trained on gpu, but i wanna predict it on cpu

austere swift Jan 6, 2021, 2:11 PM

#

Ohhh ok

#

How exactly did you transpose it?

manic granite Jan 6, 2021, 2:13 PM

#

transpose what???

#

I mean, the model is on pytorch

#

so i used pytorch2keras

#

i can share u the colab

#

https://colab.research.google.com/drive/19-S9kAhZptR8Xh5Pcz00ykB3jWn73HC2

Google Colaboratory

#

u have to download the weights from the pytorch model from

#

https://github.com/NathanUA/U-2-Net

GitHub

NathanUA/U-2-Net

The code for our newly accepted paper in Pattern Recognition 2020: "U^2-Net: Going Deeper with Nested U-Structure for Salient Object Detection." - NathanUA/U-2-Net

austere swift Jan 6, 2021, 2:34 PM

#

manic granite So if i use numpy.transpose, then the model cries because the recieved shape isn...

Here you said you tried to transpose it

#

How did you transpose it that made it not like the shape?

manic granite Jan 6, 2021, 2:40 PM

#

with transpose

#

numpy

#

(2,0,1)

austere swift Jan 6, 2021, 2:41 PM

#

It should be 0231 right?

#

Because you’re turning NCHW to NHWC, so the second element is being moved to the last

#

So 0231

manic granite Jan 6, 2021, 2:46 PM

#

it doesnt matter anyway

#

just read the error i get

#

So, keras model cries because the image are not on NHWC (batch, heigth, width, channels). My images are with channels first

#

So if i use numpy.transpose, then the model cries because the recieved shape isnt the expected

#

It cries for one thing or another

austere swift Jan 6, 2021, 2:50 PM

#

then transpose the images too

#

lemon_swag

manic granite Jan 6, 2021, 2:51 PM

#

?????

#

what do u think i am transposing?

#

xdddddddddddddddd

austere swift Jan 6, 2021, 2:52 PM

#

i mean change the model input, and transpose the images

manic granite Jan 6, 2021, 2:52 PM

#

austere swift i mean change the model input, *and* transpose the images

yeah, but changing model implies retraining it

#

and i was trying to avoid this

austere swift Jan 6, 2021, 2:53 PM

#

oh yeah

manic granite Jan 6, 2021, 2:53 PM

#

since i already have weights

#

170 Mb

#

file size

#

i was trying to reuse them

#

and also idk the data set the model was trained with

austere swift Jan 6, 2021, 2:53 PM

#

also, why not just use pytorch instead of using keras?

#

that would avoid all of this

manic granite Jan 6, 2021, 2:54 PM

#

cuz the output from this model is gonna be the input from another one and this second one is of my own, written on keras

austere swift Jan 6, 2021, 2:55 PM

#

that doesnt really matter, since the output can just be converted from a pytorch tensor to a numpy array and then moved as the input to the keras model

manic granite Jan 6, 2021, 2:55 PM

#

ye i know but i just wanna use keras on everything and not load too many libraries

#

cuz i am planing to make a discord bot with this 😛

#

so if i could somehow make it working on keras......

royal thunder Jan 6, 2021, 4:04 PM

#

guys i wanna learn ML

#

I dont know what tools is there to learn for starting

lapis sequoia Jan 6, 2021, 5:06 PM

#

royal thunder I dont know what tools is there to learn for starting

Check the pin comments

coral walrus Jan 6, 2021, 5:36 PM

#

heya,

missing_values = ['?', '--']
rows = ['X', 'Y', 'Z']

df = pd.DataFrame({
    'A': [np.nan, np.nan, 0.979],
    'B': [np.nan, 0.478, 0.029],
    'C': [1.123, np.nan, 0.720],
    'D': [0.938, 0.498, np.nan],
    'E': [1.073, 0.766, 0.984]}, na_values=missing_values, index=rows)

df

is it no longer possible to replace NaN with custom missing values when creating a dataframe using na_values=''?

#

nvm, I guess I use df.fillna()

barren bluff Jan 6, 2021, 7:03 PM

#

Hey what is a cost function and how does it work exactly?

manic granite Jan 6, 2021, 7:08 PM

#

@velvet thorn i am still waiting for you to tell me what should i reshape to make it work

summer vine Jan 6, 2021, 8:11 PM

#

Is it possible to get the mathplotlib pyplot figure image object?

nova plover Jan 6, 2021, 9:29 PM

#

for ssml, what is the best way to edit individual words pitch, i know about the contour but it doesn't seem to be working very well

summer vine Jan 6, 2021, 10:20 PM

#

summer vine Is it possible to get the mathplotlib pyplot figure image object?

figured it out lol

torpid sphinx Jan 6, 2021, 10:25 PM

#

Hi, when wanting to remove outliers from a boxplot, does “showfliers=False” only remove the outliers from the boxplot or from the data itself?
I would like to remove the outliers from the data itself

nova plover Jan 6, 2021, 10:35 PM

#

guys i made a question on stack overflow

stable otter Jan 6, 2021, 10:37 PM

#

Im still deciding between which one to use for deep learning projects. Tensorflow or pytorch
so advice would be great

odd lion Jan 6, 2021, 10:43 PM

#

stable otter Im still deciding between which one to use for deep learning projects. Tensorflo...

Both are excellent, these are pretty good comparisons: https://realpython.com/pytorch-vs-tensorflow/#pytorch-vs-tensorflow-decision-guide https://builtin.com/data-science/pytorch-vs-tensorflow Really it mostly comes down whatever your employer or any mentors/friends use so you can get the most help

odd lion Jan 6, 2021, 10:45 PM

#

barren bluff Hey what is a cost function and how does it work exactly?

Basically how incorrect the model is when trying to estimate y from X, you want to minimize it, the most common way is gradient descent

odd lion Jan 6, 2021, 10:45 PM

#

summer vine Is it possible to get the mathplotlib pyplot figure image object?

I think you mean you want to save it? Just do plt.savefig

odd lion Jan 6, 2021, 10:46 PM

#

torpid sphinx Hi, when wanting to remove outliers from a boxplot, does “showfliers=False” only...

You'd have to remove the outliers before you even pushed the data to boxplot

velvet thorn Jan 6, 2021, 11:09 PM

#

stable otter Im still deciding between which one to use for deep learning projects. Tensorflo...

honestly both work; they just have different styles

#

there are technical differences that relate to the computational graph, but not really something you need to worry about at this point?

#

starting on one at all would be more important IMO

stable otter Jan 6, 2021, 11:10 PM

#

which one is better for just some light weight projects

velvet thorn Jan 6, 2021, 11:10 PM

#

stable otter which one is better for just some light weight projects

both have high enough levels of abstraction that it doesn't really matter?

#

the differences are more obvious when you enter the depths of DL

#

you can create workable models very quickly with either

stable otter Jan 6, 2021, 11:11 PM

#

ok thx

velvet thorn Jan 6, 2021, 11:12 PM

#

a cost function represents the "cost" of doing something.

#

if you're doing supervised learning, it can be used to calculate how different your predictions are from actual data.

#

the concept also applies to other forms of ML

stable otter Jan 6, 2021, 11:13 PM

#

are there functions for reinforement learning/q-learning or is it jus python logic

velvet thorn Jan 6, 2021, 11:14 PM

#

barren bluff Hey what is a cost function and how does it work exactly?

for example, reinforcement learning, where a cost function can be used to assess which action to take, given a particular state (though in this case perhaps a reward function is more common)

#

you can just think of a reward function as the negative of the cost function though

#

(you want to maximise reward and minimise cost)

velvet thorn Jan 6, 2021, 11:14 PM

#

stable otter are there functions for reinforement learning/q-learning or is it jus python log...

what do you mean

stable otter Jan 6, 2021, 11:15 PM

#

like if you wanna make a model in tensorflow you'd say keras.Sequential(example)

#

what would u do for reinforcement learning

velvet thorn Jan 6, 2021, 11:16 PM

#

ah, okay

#

so your question is more...what builtin primitives are there in TF/PyTorch for reinforcement learning?

#

long story short

#

it's a bit more involved

#

you'd probably need to write some more basic tensor operations

#

but there are libraries which provide appropriate abstractions on top of lower-level TF

#

I believe the TF website has examples

stable otter Jan 6, 2021, 11:20 PM

#

ok thx

manic granite Jan 6, 2021, 11:45 PM

#

@velvet thorn so u wont tell me ur brilliant solution of just reshape? 🙂

velvet thorn Jan 6, 2021, 11:45 PM

#

manic granite <@!171929073063297024> so u wont tell me ur brilliant solution of just reshape? ...

honestly your attitude makes me not want to help you

#

so

#

take it that I was wrong and don't know how to

manic granite Jan 6, 2021, 11:46 PM

#

that much ego that u cant admit u were wrong? like wow we all make mistakes

wintry nacelle Jan 6, 2021, 11:57 PM

#

Simple question, I'm following the DCGAN tutorial though I am using it as a basis to learn more about GANs generally.
There's a step where I reshape the train_images numpy array from (60000, 28, 28) to (60000, 28, 28, 1). The tutorial doesn't explain why this is done beyond it being a preprocessing operation. Why add this 4th dimension?

#

oh

#

It's so that each element in the 3rd dimension is an array instead of a number unto itself

#

Feel a bit silly now

ancient lichen Jan 7, 2021, 2:51 AM

#

anyone know how to get tensorflow to work on an m1 mac?

west junco Jan 7, 2021, 2:52 AM

#

where is the best place to learn datastructures and algorithms as a beginner?

velvet thorn Jan 7, 2021, 3:20 AM

#

manic granite that much ego that u cant admit u were wrong? like wow we all make mistakes

think what you want

velvet thorn Jan 7, 2021, 3:20 AM

#

wintry nacelle It's so that each element in the 3rd dimension is an array instead of a number u...

basically, yes

#

you'll see this pattern a lot in many libraries because there's a need to disambiguate

#

if you look at classic ML, for example, which tends to take in tabular (2D) features

#

you can see that if you only had one sample, or one feature, one of the axes would have length 1

#

if the models accepted 1D data instead of 2D data with one singleton (length 1) axis, then they wouldn't be able to tell between those two cases

ancient lichen Jan 7, 2021, 3:22 AM

#

gm you have any idea about my question?

velvet thorn Jan 7, 2021, 3:22 AM

#

ancient lichen gm you have any idea about my question?

not a Mac user, sorry

#

but what's the problem

ancient lichen Jan 7, 2021, 3:22 AM

#

it always says 'Process finished with exit code 132 (interrupted by signal 4: SIGILL)'

velvet thorn Jan 7, 2021, 3:22 AM

#

by M1 Mac

#

you mean the one with the new chip?

ancient lichen Jan 7, 2021, 3:22 AM

#

yes

velvet thorn Jan 7, 2021, 3:23 AM

#

ancient lichen it always says 'Process finished with exit code 132 (interrupted by signal 4: SI...

hard to say TBH

#

I have little experience with Macs

ancient lichen Jan 7, 2021, 3:23 AM

#

I've looked it up and people have various solutions including installing apple's forked version, but for some reason when I used that tensorflow just died

#

like it had no methods or anything

wintry nacelle Jan 7, 2021, 3:23 AM

#

Hey GM I have a question that I could probably solve myself but it's currently giving me a nightmare since I have no clue how to do it.
I need to convert a bunch of .dds files into a nested numpy array containing RGBA data

#

I have no clue how to do that

ancient lichen Jan 7, 2021, 3:23 AM

#

it has to do with them being on an ARM chip now

velvet thorn Jan 7, 2021, 3:23 AM

#

could be a Mac-specific thing or something to do with your specific installation

#

can't help you there, sorry

velvet thorn Jan 7, 2021, 3:23 AM

#

wintry nacelle Hey GM I have a question that I could probably solve myself but it's currently g...

what's a dds file

ancient lichen Jan 7, 2021, 3:23 AM

#

ight, thanks anyways

velvet thorn Jan 7, 2021, 3:23 AM

#

ancient lichen ight, thanks anyways

maybe try TF forums

wintry nacelle Jan 7, 2021, 3:24 AM

#

velvet thorn what's a dds file

A file type that GPUs can handle really well. PIL can read it just fine, though I do need to correct the image

velvet thorn Jan 7, 2021, 3:24 AM

#

and what do you mean "nested" array

#

arrays should always be flat

#

unless you mean multidimensional?

ancient lichen Jan 7, 2021, 3:24 AM

#

velvet thorn maybe try TF forums

what's that?

wintry nacelle Jan 7, 2021, 3:24 AM

#

velvet thorn unless you mean multidimensional?

That's what I meant

velvet thorn Jan 7, 2021, 3:24 AM

#

wintry nacelle That's what I meant

the TensorFlow forums

velvet thorn Jan 7, 2021, 3:24 AM

#

wintry nacelle That's what I meant

okay, so you've loaded the image with PIL and have an array, I presume

#

what do you want to do with it

ancient lichen Jan 7, 2021, 3:25 AM

#

velvet thorn the TensorFlow forums

is that a site?

velvet thorn Jan 7, 2021, 3:25 AM

#

ancient lichen is that a site?

I believe there are official TF community forums

#

or you can try SO

wintry nacelle Jan 7, 2021, 3:25 AM

#

velvet thorn what do you want to do with it

Send it through preprocessing (color and size correction basically) so that the entire array is ready for my DCGAN (I have already set it up)

velvet thorn Jan 7, 2021, 3:25 AM

#

wintry nacelle Send it through preprocessing (color and size correction basically) so that the ...

do you know what transformations you intend to apply?

#

what specific problem do you have then

wintry nacelle Jan 7, 2021, 3:26 AM

#

The multidimensional numpy array itself

#

I don't know how to make it or fill the data properly

velvet thorn Jan 7, 2021, 3:26 AM

#

hm

#

I am confused

#

thought you said you loaded it successfully

wintry nacelle Jan 7, 2021, 3:26 AM

#

I can load each individal file correctly using glob and PIL

#

I can also convert it into numpy array data that I can then have read back

velvet thorn Jan 7, 2021, 3:27 AM

#

what does each individual file represent?

wintry nacelle Jan 7, 2021, 3:27 AM

#

A single graphic

velvet thorn Jan 7, 2021, 3:27 AM

#

okay

wintry nacelle Jan 7, 2021, 3:27 AM

#

Lemme send you an example

velvet thorn Jan 7, 2021, 3:27 AM

#

go on

arctic wedgeBOT Jan 7, 2021, 3:27 AM

#

Hey @wintry nacelle!

It looks like you tried to attach file type(s) that we do not allow (.dds). We currently allow the following file types: .3gp, .3g2, .avi, .bmp, .gif, .h264, .jpg, .jpeg, .mkv, .mov, .mp4, .mpeg, .mpg, .png, .tiff, .wmv, .svg, .psd, .ai, .aep, .xcf, .mp3, .wav, .ogg, .webm, .webp, .flac, .afdesign, .m4a, .csv.

Feel free to ask in #community-meta if you think this is a mistake.

wintry nacelle Jan 7, 2021, 3:27 AM

#

gdmit

#

Wait I know how to get a PNG version

#

actually know what screw it

#

All the images I'm working with can be found in a PNG format here: https://hoi4.paradoxwikis.com/Category:National_focus_icons

Category:National focus icons

velvet thorn Jan 7, 2021, 3:28 AM

#

I mean

#

okay, I can understand that you load them into a numpy array

wintry nacelle Jan 7, 2021, 3:28 AM

#

I'm in my learning phase so I'm just experimenting basically

velvet thorn Jan 7, 2021, 3:28 AM

#

but what next

wintry nacelle Jan 7, 2021, 3:28 AM

#

Do preprocessing, so color correction and adding padding to the image so that they are all 100x100 (they are not consistently sized)

velvet thorn Jan 7, 2021, 3:28 AM

#

what colour correction algorithm do you want to apply?

wintry nacelle Jan 7, 2021, 3:28 AM

#

Then sending it into the model should be easy

velvet thorn Jan 7, 2021, 3:28 AM

#

and how do you want to pad the images?

#

I'm assuming

#

you want to roughly centre them

#

in the padding?

wintry nacelle Jan 7, 2021, 3:29 AM

#

velvet thorn what colour correction algorithm do you want to apply?

I'm not struggling with that
It's more about how the format is not being read as RGB when it should (or perhaps the other way around), so I need to convert the data first

#

Right now I'm getting strange colors which I'm putting down to the RGBA values being mixed up in some way

velvet thorn Jan 7, 2021, 3:30 AM

#

OKAY

#

it would have been helpful to know that

#

anyway

#

so what you really mean is

#

the image isn't being read properly?