🔒 what is pandas and numpy and plt...?

[short obsidian]

i decided to collect history data of the skyblock bazaar they got an api but it can only tell the current state
what i want is to get all items
extract a few informations i need and filter the really cheap/really pricy items
later i want to draw some graphs based on it and calculate some statistics
i heard that for large data like this people use pandas numpy and plt
how should i start?
(never or only slightly have i used any of them)

[kind wagon]

@short obsidian

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[reef badger]

numpy is for bulk computation on arrays.
pandas is for this with columnar data (like a CSV or spreadsheet), uses a lot of numpy
You might look at seaborn for plotting statistics, though pandas and numpy both have plotting (they use matplotlib by default).

[cunning bough]

additionally, plt probably is matplotlib.pyplot

[short obsidian]

yes i used that a bit just forgot how it works

[reef badger]

Easiest way can often be from a pandas dataframe, it's got a few methods to plot common things.

[cunning bough]

depending on how much data you're expecting, you might also be fine with none of that and just process in good ol' vanilla python

[short obsidian]

i have bot much data per item but prob between 1k and 10k items i want to monitor and keep record of

[short obsidian]

i got a dict like

"products": {
"INK_SACK:3": {
"product_id": "INK_SACK:3",
"sell_summary": [
{
"amount": 20569,
"pricePerUnit": 4.2,
"orders": 1
},
{
"amount": 140326,
"pricePerUnit": 3.8,
"orders": 2
}
],
"buy_summary": [
{
"amount": 640,
"pricePerUnit": 4.8,
"orders": 1
},
{
"amount": 640,
"pricePerUnit": 4.9,
"orders": 1
},
{
"amount": 25957,
"pricePerUnit": 5,
"orders": 3
}
],
"quick_status": {
"productId": "INK_SACK:3",
"sellPrice": 4.2,
"sellVolume": 409855,
"sellMovingWeek": 8301075,
"sellOrders": 11,
"buyPrice": 4.99260315136572,
"buyVolume": 1254854,
"buyMovingWeek": 5830656,
"buyOrders": 85
}
}
}

how do i get
productId
sell and buy price
min of the 2 moveing weeks
and put it in a pandas dataframe?

i will always have the same few dosen productIds so best if i only store it once and i want to order it by date best if i also add a date to the data

[summer oracle]

!e

import pandas as pd

d = {"products": {
"INK_SACK:3": {
"product_id": "INK_SACK:3",
"sell_summary": [
{
"amount": 20569,
"pricePerUnit": 4.2,
"orders": 1
},
{
"amount": 140326,
"pricePerUnit": 3.8,
"orders": 2
}
],
"buy_summary": [
{
"amount": 640,
"pricePerUnit": 4.8,
"orders": 1
},
{
"amount": 640,
"pricePerUnit": 4.9,
"orders": 1
},
{
"amount": 25957,
"pricePerUnit": 5,
"orders": 3
}
],
"quick_status": {
"productId": "INK_SACK:3",
"sellPrice": 4.2,
"sellVolume": 409855,
"sellMovingWeek": 8301075,
"sellOrders": 11,
"buyPrice": 4.99260315136572,
"buyVolume": 1254854,
"buyMovingWeek": 5830656,
"buyOrders": 85
}
}
}}
df = pd.DataFrame(columns=["product id", "sell price", "buy price", "min 2 weeks moving"])
status = d["products"]["INK_SACK:3"]["quick_status"]
product_id, sell_price, buy_price, min_of_2_weeks = status["productId"], status["sellPrice"], status["buyPrice"], min(status["sellMovingWeek"], status["buyMovingWeek"])
df.loc[0] = [product_id, sell_price, buy_price, min_of_2_weeks]
print(df)

[kind wagon]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

001 |    product id  sell price  buy price  min 2 weeks moving
002 | 0  INK_SACK:3         4.2   4.992603             5830656

[summer oracle]

something like that?

[short obsidian]

prob ye but i need to access it by time too

(and got iter the product but ill do that part)

and how would i append a row?
do i have to find where the last index is?

[summer oracle]

if you get all the data into a list, you can create the dataframe at the end in one go. I was just providing an example

[short obsidian]

i need to remove the eccess data so i already have a loop anyways

[summer oracle]

cool, you can provide the list into pd.DataFrame when you create it and it should make the df for yo

[short obsidian]

ill do that than but i still need to append the updates

bc ill collect data every 20 or so minutes to build up history i can graph

[summer oracle]

are you updating one line at a time?

you could create a dataframe with the same columns and concatenate it with the main one

[short obsidian]

like maybe 1k lines at a time based on how many products i want to monitor

[short obsidian]

seems a good method for me is it just +=?

[summer oracle]

right, so you run the loop to make a df of 1k lines, then concatenate with the previous

pd.concat()

[short obsidian]

ah thx

def weed(product: dict, date: datetime) -> list[str, int]:
    Id = product["product_id"]
    summary = product["quick_status"]
    sell = summary["sellPrice"]
    buy = summary["buyPrice"]
    movement = min(summary["sellMovingWeek"], summary["buyMovingWeek"])
    row = [Id, date, sell, buy, movement]
    return row

def simplify(products: dict, selection: set[str], date: datetime) -> pd.DataFrame:
    products = [weed(product,date) for product in products.values() if product["product_id"] in selection]
    df = pd.DataFrame(
        data=products,
        columns=["Id", "date", "sell", "buy", "movement"], 
        index=["Id", "date"]
    )
    return df

looks ok?

im pretty sure i could do some numpy instead of that for loop
ik about vectorify but how do i implement the if?

[summer oracle]

you could make a dataframe with all the products and then drop the rows where the product id is not in selection

[short obsidian]

nice so i just make an anti selection instead of a selection and done?

[summer oracle]

there is a Series.isin() method you can use with boolean indexing on the dataframe

df = df[df['column'].isin(selection)]
``` something like that

[short obsidian]

would it be faster if i do it on the np array does np have that funcion?

[summer oracle]

pandas is built on top of numpy and is pretty fast

[short obsidian]

i was just thinking why add all elements to a list than remove a few if u can just not add them in the first place

[summer oracle]

the main bottleneck will probably be looping over the dictionary. You could try pd.json_normalise() on the full dictionary to turn it into a workable dataframe, if you can get the 4 columns you need to come out nicely, then everything gets done in milliseconds

[summer oracle]

oh right I see, you're correct, I just misread your simplify function at first

[short obsidian]

i was thinking of np.vectorize

@np.vectorize
def weed(product: dict, date: datetime) -> list[str, int]:
    Id = product["product_id"]
    summary = product["quick_status"]
    sell = summary["sellPrice"]
    buy = summary["buyPrice"]
    movement = min(summary["sellMovingWeek"], summary["buyMovingWeek"])
    row = [Id, date, sell, buy, movement]
    return row

def simplify(products: dict, selection: set[str], date: datetime) -> pd.DataFrame:
    products = weed(np.array(products.values()))

    df = pd.DataFrame(
        data=products,
        columns=["Id", "date", "sell", "buy", "movement"], 
        index=["Id", "date"]
    )
    df = df[df['Id'].isin(selection)]
    return df

[summer oracle]

give it a go

[short obsidian]

how do i save the df?

[summer oracle]

df.to_csv() or df.to_excel()

[short obsidian]

i miss the date arg from weed how do i fix that?

i would first take a partial than vectorize it?

is there a better way?

[summer oracle]

I don't see a date in your dictionary, what date do you want

[short obsidian]

the one given to simplify

i need to know when a price was recorded

how do i turn the dict_values object to an np array?

[summer oracle]

well... not sure how to do that nicely, which is why I suggested pd.json_normalise

[short obsidian]

that is what i want to do if i see correctly nice

how do i specify the paths?

bc thair depth is more than 1

and how do i rename the columns?

[summer oracle]

path also takes a list of strings

[short obsidian]

and how do i do the min thing

[summer oracle]

once you have the dataframe, create a new column where you calculate it

[short obsidian]

f = np.vectorize(partial(weed, date=date))
products = f(np.array(list(products.values())))
this would work but returns (1441, 1) and i need a return shape of (1441, 5)

[short obsidian]

i managed to get it to work

what do i pass to excel writer?

[summer oracle]

just filepath, including filename, where you want to have it saved

it says ExcelWriter because you can make these special objects and work with them differently, so it's just something extra, but treat it as normal save function

[short obsidian]

df.to_excel(excel_writer="data") says im missing an engine/filetype

[cunning bough]

(side note: np.vectorize is a convenience function and does not provide any performance benefits)

[summer oracle]

"data.xlsx" or whatever is the correct suffix for excel

[short obsidian]

i thought it does loops in some lower level or uses gpu or something to speed it up a bit

[summer oracle]

using gpu is difficult, you wouldn't expect to suddenly have your gpu be used like that

[short obsidian]

yay i did it

[cunning bough]

nope, np.vectorize is just a python loops wrapper

[short obsidian]

import requests
from typing import Callable
import pandas as pd
import numpy as np
from datetime import datetime
from functools import partial

def get_all() -> dict:
    return requests.get("https://api.hypixel.net/v2/skyblock/bazaar").json()["products"]

def weed(product: dict, date: datetime) -> list[str, int]:
    Id = product["product_id"]
    summary = product["quick_status"]
    sell = summary["sellPrice"]
    buy = summary["buyPrice"]
    movement = min(summary["sellMovingWeek"], summary["buyMovingWeek"])
    row = [Id, date, sell, buy, movement]
    return np.array(row)

def simplify(products: dict, selection: set[str], date: datetime) -> pd.DataFrame:
    f = np.vectorize(partial(weed, date=date))
    products = list(products.values())
    products = f(np.array(products, dtype=dict))
    products = np.stack(products)
    print(products.shape)
    #pd.json_normalize(        products,        record_path = [["quick_status", "sellPrice", buy]]    )
    df = pd.DataFrame(
        data=products,
        columns=["Id", "date", "sell", "buy", "movement"], 
        #index=["Id", "date"],
    )
    #df = df[df['Id'].isin(selection)]
    return df
        

def check(product: dict) -> bool:
    pass

def make_monitor_list(f: Callable[[dict], bool]) -> list[str]:
    return list(filter(f, get_all()))

data = get_all()
df = simplify(data, set(), datetime.now())
df.to_excel(excel_writer="data.xlsx")

now i gotta clean this up and next is to make a filter list

[short obsidian]

than ill go back to a list comp

but should i check in a set in the for or should i drop the rows?
is set O(1) or O(logn)?

[cunning bough]

set/dicts in python are backed by hash tables

[short obsidian]

how do i make it so i can get items from the dataframe by Id or by date?

[cunning bough]

you can do df.loc[row_mask]
e.g. df.loc[ df['price'] == 1234 ] gets you item(s) whose price is 1234

[short obsidian]

nice

what is group by?
iw used it once but dont quite remember / get what it does
would it make sense for me to group by id?

[summer oracle]

if you make productid an index, then you can do df['INk_SACK:3'] and I think that returns the corresponding row

[short obsidian]

there are multiple rows with the same Id or the same date
(but never 2 with both the same)

[summer oracle]

groupby is for when you need to aggregate data, so like you might want to know price of something every 3 days, then you can put the prices into groups that contains data every 3 days

[cunning bough]

basically SQL group by

[short obsidian]

idk sql im just a starter in this

[cunning bough]

processing data item-by-item is prob not efficient, tho ig how you do it depends on your goal

[short obsidian]

im trying to get a historic data of the prices and demand
so i can make some bots that tell me the best flip

[cunning bough]

then what Jeremy said above
or another example: you have a productType column and want to find the average price of each product, then something something df.groupby('productType').mean()

[short obsidian]

so groupby is used before calculations but not stored as groups

[summer oracle]

you can have it make you groups if you .reset_index() on the groupby object, but that's not the primary focus

[short obsidian]

i can save to exel but can i load from it or do i have to use csv for that?

[summer oracle]

you can save and load from either

[short obsidian]

how do i load from excel?

[summer oracle]

pd.read_excel()

[short obsidian]

i made this thing and the excel it makes looks wierd

[kind wagon]

Click here to see this code in our pastebin.

[short obsidian]

it randomly added an other col
the indexing keeps restarting this extra col misses the numbers somethimes

[summer oracle]

use index=False when saving

[short obsidian]

nice

now ill find somewhere to let it run and work on drawing graphs

[kind wagon]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.