🔒 Making code faster - Current Solution uses a very very long list | Python | Page 1

frosty vault May 16, 2024, 2:14 AM

#

I would like to calculate the standard deviation of a dataset, which involves subtracting the value in the dataset with the mean.

I'm not sure how to calculate each value, find its mean, and then go back to all of the other values to find the difference without creating a very long list with every single value.

worn shoreBOT May 16, 2024, 2:14 AM

#

frosty vault I would like to calculate the standard deviation of a dataset, which involves su...

#

@frosty vault

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

frosty vault May 16, 2024, 2:16 AM

#

import random
import matplotlib.pyplot as plt
import pandas as pd
import math
import scipy
from scipy.stats import norm
import numpy as np

binSize = 3
sampleSize = 5
numberOfTrials = 100000000

data = [
    122,
    114,
    94,
    88,
    92,
    88,
    96,
    108,
    122,
    108,
    98,
    96,
    70,
    66,
    86,
    90,
    96,
    96,
    88,
    104,
    104]

histogramList = []

maximum = max(data)
minimum = min(data)

maximum += binSize - (maximum - minimum) % binSize

lowRange = minimum
highRange = minimum + binSize - 1

meanList = []
for a in range(int((maximum-minimum)/(binSize-1))):
    histogramList.append([lowRange, highRange, 0])

    lowRange += binSize -1
    highRange += binSize-1

totalMean = 0
for b in range(numberOfTrials):
    sum = 0
    for i in range(sampleSize):
        value = data[random.randint(0,len(data)-1)]
        sum += value
    mean = sum / sampleSize
    totalMean += mean
    meanList.append(mean)

    for c in histogramList:
        lowRange, highRange, counter = c
        if mean >= lowRange and mean <= highRange:
            c[2] += 1
            break


xAxis = []
yAxis = []



variance = 0
finalMean = totalMean/numberOfTrials
for e in meanList:
    variance += (e-finalMean)**2

variance = variance/(numberOfTrials )
stdDev = math.sqrt(variance)


for d in histogramList:
    xAxis.append(f"{d[0]} - {d[1]}")
    yAxis.append(d[2])

df = pd.DataFrame({"Sample Mean": xAxis, 'Frequency': yAxis})

plt.bar(df['Sample Mean'], df['Frequency'], width=0.8)
plt.xticks(rotation = 90)
x = ((np.linspace(minimum, maximum, 100)))

pdf = norm.pdf(finalMean, finalMean, stdDev*math.sqrt(1))


save = 0
saveNumber = 0
for h in range(len(histogramList)-1):
    if histogramList[h][0] <= finalMean and histogramList[h][1] >= finalMean:
        saveNumber = h
        # save = (histogramList[h][0] + histogramList[h][1])/2
        save = histogramList[h][2]

factor = save / pdf

plt.show()

odd fractal May 16, 2024, 2:32 AM

#

any reason to not ```py
import statistics
statistics.stdev(data)

#

your code seems to be doing more than what your initial question suggests. what's the overall goal of the code?

frosty vault May 16, 2024, 3:05 AM

#

odd fractal your code seems to be doing more than what your initial question suggests. what'...

The overall goal of the code is to create a sampling distribution by taking five random values from the data set, finding the mean, and the plotting it on a graph

#

According to Central limit theorem, this graph should resemble a normal distribution curve

#

In order for me to overlay a normal distribution curve, I need the standard deviation (the square root of the variance, which calculating is what is requiring this long list)

frosty vault May 16, 2024, 3:07 AM

#

odd fractal any reason to not ```py import statistics statistics.stdev(data) ```?

How the code current works is:

Choose five random numbers from data set -> find mean -> append mean to a list -> Find the average of all of these individual means -> use this average to find the standard deviation by comparing it to all of the individual means

#

Here's the formual for variance, what I am hoping to calculate

odd fractal May 16, 2024, 3:08 AM

#

oh, do you just want to eliminate the meanList?

frosty vault May 16, 2024, 3:09 AM

#

Not necessarily. I just want a process that takes up less memory (because if I find 10 million means, then I have a list that is 10 million items long), and is also faster

#

because it takes quite a while to process

odd fractal May 16, 2024, 3:09 AM

#

how many trials are you running?

#

ah, numberOfTrials = 100000000

#

so quite a few!

frosty vault May 16, 2024, 3:10 AM

#

Yes!

#

The more, the better!

odd fractal May 16, 2024, 3:13 AM

#

so one simple optimization

#

there's not that many distinct mean values

#

# at top of file
from collections import defaultdict

# instead of meanList = []
meanCounts = defaultdict(int)

# instead of meanList.append(mean)
meanCounts[mean] += 1

# instead of for loop over meanList
for mean, count in meanCounts.items():
    variance += ((mean-finalMean)**2) * count

frosty vault May 16, 2024, 3:17 AM

#

odd fractal ```py # at top of file from collections import defaultdict # instead of meanLis...

Oh! interesting

#

Could you tell me how defaultdict works? I'm not familiar with the library

odd fractal May 16, 2024, 3:18 AM

#

are you familiar with regular dicts

frosty vault May 16, 2024, 3:18 AM

#

not much

odd fractal May 16, 2024, 3:18 AM

#

in short a default dictionary is a dictionary for which you can specify a default value if a key doesn't exist

#

!e ```py
from collections import defaultdict
def default_value():
return 7
x = defaultdict(default_value)
print(x["foo"])

worn shoreBOT May 16, 2024, 3:19 AM

#

@odd fractal :white_check_mark: Your 3.12 eval job has completed with return code 0.

odd fractal May 16, 2024, 3:19 AM

#

so this optimization bounds your memory usage to the size of data, rather than the number of trials

#

it'll still run for a very long time, but in theory it will never OOM, if that's the issue you were running into previously

frosty vault May 16, 2024, 3:21 AM

#

Ohh ok

#

I don't think I completely understand here.

so with defaultdict, you have a default value for a key that doesn't exist

#

Are you putting this value as 0? And then now it becomes a key within the dictionary

#

So that when referred to again, if it is a valid key when put into defaultdict(int) , it just adds one ?

odd fractal May 16, 2024, 3:23 AM

#

yeah, in this context int is a function which returns 0

#

and then yes, when we access a value that may not exist, it inserts 0 into the dict

#

then we add 1 to that

#

you should also be able to simplify some of the logic in your code loop

#

sum = 0
for i in range(sampleSize):
    value = data[random.randint(0,len(data)-1)]
    sum += value

vs

total = sum(random.choices(data, k=sampleSize))

#

and the latter should be slightly faster, since it's largely written in C

#

there's also random.sample if you want a sample without duplicates

#

oh wow

#

it actually finished executing

frosty vault May 16, 2024, 3:27 AM

#

odd fractal there's also `random.sample` if you want a sample without duplicates

Without duplicates/

#

?

frosty vault May 16, 2024, 3:27 AM

#

odd fractal it actually finished executing

oh! that's quite a while haha

#

Its okah, i can live with that

#

i'm on less than a GB left of storage so this memory save helps

odd fractal May 16, 2024, 3:28 AM

#

frosty vault Without duplicates/

if you want to take a random subset of the elements

#

e.g. data = [1, 2, 3] sample k=2 might be [3, 1] but the existing code could produce [1, 1]

#

which may be fine for your purposes

#

Captura_de_pantalla_2024-05-15_a_las_11.33.19_p.m..png

frosty vault May 16, 2024, 3:52 AM

#

Ohhh

#

I am okay with having [1,1]

frosty vault May 16, 2024, 3:53 AM

#

odd fractal

👍

#

Thank you so much for your help, I'll try adding your ideas

small ferry May 16, 2024, 4:25 AM

#

frosty vault ```py import random import matplotlib.pyplot as plt import pandas as pd import m...

you shouldn't be looping if you use numpy

small ferry May 16, 2024, 4:30 AM

#

frosty vault How the code current works is: Choose five random numbers from data set -> fin...

use np.random to choose k sets of 5 data points, np.mean to find all k means of each set, np.sum these to get a cumulative value (if memory allows, you can choose very big k)
repeat the above until you're satisfied, divide the cumulative value by how many means you added together to get the average

small ferry May 16, 2024, 4:32 AM

#

frosty vault Here's the formual for variance, what I am hoping to calculate

iirc, that summation in the numerator expands to something like (sum(x**2) - n*mu**2) so you don't need to remember all the values

worn shoreBOT May 16, 2024, 5:32 AM

#

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 Making code faster - Current Solution uses a very very long list