🔒 Issue getting code to recognize Volume | Python | Page 1

sinful slate Mar 30, 2024, 2:15 AM

#

Okay so here's my code so far
import time
import datetime
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import ssl
from scipy.stats import skew

ssl._create_default_https_context = ssl._create_unverified_context

tables = pd.read_html('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')

symbolslist=tables[0]['Symbol'].to_list()

my_data=yf.download(symbolslist, start=None, end=None, actions=False)

open_high_data = my_data[['Open','High','Close','Volume']]

open_high_data.index = pd.to_datetime(open_high_data.index)

between = open_high_data.loc['2010-01-01':'2023-12-31']

stacked_data = between.stack()

rearranged_data = stacked_data.reset_index()

rearranged_data.columns = ['Date', 'Symbol', 'Open', 'High', 'Close',' Volume']

print(rearranged_data)

rearranged_data['Daily_Return'] = rearranged_data.groupby('Symbol')['Open'].pct_change()

monthly_avg_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M'))['Daily_Return'].mean()

monthly_std_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].std()

monthly_skew_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].apply(skew)

rearranged_data['Avg_Daily_Return'] = rearranged_data.set_index(['Symbol','Date']).index.map(monthly_avg_daily_returns)

rearranged_data['Std_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_std_daily_returns)

rearranged_data['Skewness_Daily_Return'] = rearranged_data.set_index(['Symbol','Date']).index.map(monthly_skew_daily_returns)

print(rearranged_data)

All this? Working so far. Here's the part that's causing issues.

rearranged_data['Dollar_Trade_Volume'] = rearranged_data['Close'] * rearranged_data['Volume']

For whatever reason, it keeps giving me a error saying "Keyerror: Volume"

This assignment is due at midnight EST. I could really use some quick help here.

dreamy duneBOT Mar 30, 2024, 2:15 AM

#

sinful slate Okay so here's my code so far `import time` `import datetime` `import pandas as ...

#

@sinful slate

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

tribal venture Mar 30, 2024, 2:17 AM

#

I get a syntax error from that

#

try pasting it properly

#

!code

dreamy duneBOT Mar 30, 2024, 2:17 AM

#

Formatting code on Discord

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

For long code samples, you can use our pastebin.

sinful slate Mar 30, 2024, 2:17 AM

#

sorry about that

#

import time
import datetime
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import ssl
from scipy.stats import skew

#bypass certificate
ssl._create_default_https_context = ssl._create_unverified_context```

dreamy duneBOT Mar 30, 2024, 2:18 AM

#

Hey @sinful slate!

It looks like you pasted Python code without syntax highlighting.

Please use syntax highlighting to improve the legibility of your code and make it easier for us to help you.

To do this, use the following method:
```py
print('Hello, world!')
```

This will result in the following:

print('Hello, world!')```
You can **edit your original message** to correct your code block.

tribal venture Mar 30, 2024, 2:19 AM

#

is that really the complete code?

#

seems pretty short

sinful slate Mar 30, 2024, 2:20 AM

#

import time
import datetime
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import ssl
from scipy.stats import skew

ssl._create_default_https_context = ssl._create_unverified_context

tables = pd.read_html('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')

symbolslist=tables[0]['Symbol'].to_list()

my_data=yf.download(symbolslist, start=None, end=None, actions=False)

open_high_data = my_data[['Open','High','Close','Volume']]

open_high_data.index = pd.to_datetime(open_high_data.index)

between = open_high_data.loc['2010-01-01':'2023-12-31']

stacked_data = between.stack()

rearranged_data = stacked_data.reset_index()

rearranged_data.columns = ['Date', 'Symbol', 'Open', 'High', 'Close',' Volume']

rearranged_data['Daily_Return'] = rearranged_data.groupby('Symbol')['Open'].pct_change()

monthly_avg_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].mean()

monthly_std_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].std()

monthly_skew_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].apply(skew)

rearranged_data['Avg_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_avg_daily_returns)

rearranged_data['Std_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_std_daily_returns)

rearranged_data['Skewness_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_skew_daily_returns)

rearranged_data['Dollar_Trade_Volume'] = rearranged_data['Close'] * rearranged_data['Volume']

print(rearranged_data)

sinful slate Mar 30, 2024, 2:20 AM

#

tribal venture is that really the complete code?

there ya go

#

Its giving me an error on the 'volume'

#

KeyError                                  Traceback (most recent call last)
File ~\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3653, in Index.get_loc(self, key)
   3652 try:
-> 3653     return self._engine.get_loc(casted_key)
   3654 except KeyError as err:

File ~\anaconda3\Lib\site-packages\pandas\_libs\index.pyx:147, in pandas._libs.index.IndexEngine.get_loc()

File ~\anaconda3\Lib\site-packages\pandas\_libs\index.pyx:176, in pandas._libs.index.IndexEngine.get_loc()

File pandas\_libs\hashtable_class_helper.pxi:7080, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\_libs\hashtable_class_helper.pxi:7088, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'Volume'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[35], line 1
----> 1 rearranged_data['Dollar_Trade_Volume'] = rearranged_data['Close'] * rearranged_data['Volume']

File ~\anaconda3\Lib\site-packages\pandas\core\frame.py:3761, in DataFrame.__getitem__(self, key)
   3759 if self.columns.nlevels > 1:
   3760     return self._getitem_multilevel(key)
-> 3761 indexer = self.columns.get_loc(key)
   3762 if is_integer(indexer):
   3763     indexer = [indexer]

File ~\anaconda3\Lib\site-packages\pandas\core\indexes\base.py:3655, in Index.get_loc(self, key)
   3653     return self._engine.get_loc(casted_key)
   3654 except KeyError as err:
-> 3655     raise KeyError(key) from err
   3656 except TypeError:
   3657     # If we have a listlike key, _check_indexing_error will raise
   3658     #  InvalidIndexError. Otherwise we fall through and re-raise
   3659     #  the TypeError.
   3660     self._check_indexing_error(key)

KeyError: 'Volume'

#

Any thoughts?

tribal venture Mar 30, 2024, 2:23 AM

#

ok I get something very different

#


poetry  run python3 wat.py 
[*********************100%%**********************]  503 of 503 completed

2 Failed downloads:
['BF.B']: Exception('%ticker%: No price data found, symbol may be delisted (1d 1925-04-23 -> 2024-03-29)')
['BRK.B']: Exception('%ticker%: No timezone found, symbol may be delisted')
Traceback (most recent call last):
  File "/private/tmp/x/wat.py", line 18, in <module>
    open_high_data = my_data[["Open", "High", "Close", "Volume"]]
                     ^^^^^^^
NameError: name 'my_data' is not defined. Did you mean: 'y_data'?

#

you gotta post the code you're actually running

sinful slate Mar 30, 2024, 2:25 AM

#

I'm trying to. I just cut out the bits where it printed things earlier on in the process

tribal venture Mar 30, 2024, 2:25 AM

#

ok now I see it

#

poetry  run python3 wat.py 
[*********************100%%**********************]  503 of 503 completed

2 Failed downloads:
['BRK.B']: Exception('%ticker%: No timezone found, symbol may be delisted')
['BF.B']: Exception('%ticker%: No price data found, symbol may be delisted (1d 1925-04-23 -> 2024-03-29)')
/private/tmp/x/wat.py:21: FutureWarning: The previous implementation of stack is deprecated and will be removed in a future version of pandas. See the What's New notes for pandas 2.1.0 for details. Specify future_stack=True to adopt the new implementation and silence this warning.
  stacked_data = between.stack()
/private/tmp/x/wat.py:31: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  pd.Grouper(key="Date", freq="M"),
/private/tmp/x/wat.py:36: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  pd.Grouper(key="Date", freq="M"),
/private/tmp/x/wat.py:41: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  pd.Grouper(key="Date", freq="M"),
Traceback (most recent call last):
  File "/private/tmp/x/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Volume'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/x/wat.py", line 60, in <module>
    rearranged_data["Close"] * rearranged_data["Volume"]
                               ~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/private/tmp/x/.venv/lib/python3.12/site-packages/pandas/core/frame.py", line 4090, in __getitem__
    indexer = self.columns.get_loc(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/x/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'Volume'
``` right?

sinful slate Mar 30, 2024, 2:25 AM

#

yup

tribal venture Mar 30, 2024, 2:25 AM

#

ok lemme ponder

sinful slate Mar 30, 2024, 2:26 AM

#

because volume is clearly there earlier on, it's one of the data columns gathered from yahoo finance

#

And the others work fine

#

Its so weird

tribal venture Mar 30, 2024, 2:27 AM

#

well

#

I'd look into this further, but it takes like 20 seconds every time I run it, and I'm too impatient

sinful slate Mar 30, 2024, 2:27 AM

#

Sorry thats the downloading part

tribal venture Mar 30, 2024, 2:27 AM

#

if you could change it so that it caches the stuff that it downloads, and then uses that cache thenceforth -- so that the second (and subsequent) times you run it it's fast -- I'd look further

sinful slate Mar 30, 2024, 2:27 AM

#

You can break it up to the before and after gathering from yahoo info stuff

#

here

#

try this

#

import time
import datetime
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import ssl
from scipy.stats import skew

ssl._create_default_https_context = ssl._create_unverified_context

tables = pd.read_html('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')

symbolslist=tables[0]['Symbol'].to_list()

my_data=yf.download(symbolslist, start=None, end=None, actions=False)

open_high_data = my_data[['Open','High','Close','Volume']]

open_high_data.index = pd.to_datetime(open_high_data.index)

between = open_high_data.loc['2010-01-01':'2023-12-31']

stacked_data = between.stack()

rearranged_data = stacked_data.reset_index()

rearranged_data.columns = ['Date', 'Symbol', 'Open', 'High', 'Close',' Volume']

print(rearranged_data)

#


rearranged_data['Daily_Return'] = rearranged_data.groupby('Symbol')['Open'].pct_change()

monthly_avg_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].mean()

monthly_std_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].std()

monthly_skew_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].apply(skew)

rearranged_data['Avg_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_avg_daily_returns)

rearranged_data['Std_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_std_daily_returns)

rearranged_data['Skewness_Daily_Return'] = rearranged_data.set_index(['Symbol', 'Date']).index.map(monthly_skew_daily_returns)

rearranged_data['Dollar_Trade_Volume'] = rearranged_data['Close'] * rearranged_data['Volume']

print(rearranged_data)

#

same code, just split into two sections

#

So top part is gathering the info, bottom part is adding and calculating stuff with it

tribal venture Mar 30, 2024, 2:30 AM

#

why

sinful slate Mar 30, 2024, 2:31 AM

#

So that we can edit the part that's causing issues (the calculations) without having to do the download every time

tribal venture Mar 30, 2024, 2:31 AM

#

I was hoping you'd do that 🙂

#

You underestimate my laziness

sinful slate Mar 30, 2024, 2:31 AM

#

I just did

#

It's split into two parts now

tribal venture Mar 30, 2024, 2:31 AM

#

I'm interested in debugging the problem with Volume, but not with e.g. writing the caching code

sinful slate Mar 30, 2024, 2:31 AM

#

Just copy/paste and run it

tribal venture Mar 30, 2024, 2:32 AM

#

I did; it's no faster and I can't see that it'd ever be faster

sinful slate Mar 30, 2024, 2:32 AM

#

Well yeah but once you've done the download we can edit the second part and that'll be faster

tribal venture Mar 30, 2024, 2:32 AM

#

afaict, you

split your code into two pieces
stuck a "print" at the end of the first one

That's a fine start but it doesn't make the code run any faster

sinful slate Mar 30, 2024, 2:32 AM

#

...what are you using for this?

#

I'm running it in jupyer notebook

#

so each segment runs separately

#

And if the first part is cached, it applies to the rest

tribal venture Mar 30, 2024, 2:33 AM

#

I'm just plopping your code into a file and typing poetry run python3 wat.py

#

oh you're keeping the various values around as globals. That's a good idea

sinful slate Mar 30, 2024, 2:34 AM

#

yeah

#

and when the first part prints, volume is one of the values that is in the chart

tribal venture Mar 30, 2024, 2:34 AM

#

I may or may not have jupyter lying around though

sinful slate Mar 30, 2024, 2:34 AM

#

I don't know why it's not recognizing it later

tribal venture Mar 30, 2024, 2:35 AM

#

why not literally stick a print after every line?

sinful slate Mar 30, 2024, 2:35 AM

#

Would it help?

#

Just for debugging?

tribal venture Mar 30, 2024, 2:35 AM

#

presumably the "Volume" column will vanish at some point, and you'll see where it vanishes

#

yeah

sinful slate Mar 30, 2024, 2:35 AM

#

Alright, i'll give it a go

tribal venture Mar 30, 2024, 2:35 AM

#

then you'll have to say "OK, why did this line make it vanish when the other lines didn't"

sinful slate Mar 30, 2024, 2:40 AM

#

ok it seems to be working fine until it gets to here, where it gets jammed

#

monthly_skew_daily_returns = rearranged_data.groupby(['Symbol', pd.Grouper(key='Date', freq='M')])['Daily_Return'].apply(skew)

print(rearranged_data)

#

afterwards though the other parts run okay, and even print with volume

#

Nevermind it just took a moment, that part printed fine as well

#

volume is there as a column in every one

#

Its just the last line before the print that's having issues

#

maybe I could make volume its own variable? Just have volume = (however I pull it from the finance data?)

#

Nope, that didn't work

#

This is bizzare

#

@tribal venture Any thoughts?

#

I still need help

#

IT WAS A SPACE

#

A SINGLE SPACE

tribal venture Mar 30, 2024, 3:08 AM

#

heh

sinful slate Mar 30, 2024, 3:08 AM

#

' Volume'

#

wow that's dumb

#

lol

tribal venture Mar 30, 2024, 3:08 AM

#

I don't see any space

sinful slate Mar 30, 2024, 3:08 AM

#

rearranged_data.columns = ['Date', 'Symbol', 'Open', 'High', 'Close',' Volume']

#

the line where the columns were named

tribal venture Mar 30, 2024, 3:09 AM

#

ooooh

#

yeah

#

I shudda noticed that 😐

#

oh well

sinful slate Mar 30, 2024, 3:09 AM

#

I didnt notice it until I decided to plug the whole thing into chatgpt and it was like "you've got a space there bud"

#

XD

#

Well, thanks for trying!

dreamy duneBOT Mar 30, 2024, 4:09 AM

#

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 Issue getting code to recognize Volume