I sorted the JSON out based on the slightly bogus bits in the wiki and completed it with the forum threads.
#tooldev-general
pseudo ocean
does anyone know how to extract ingame assets from the content.ggpk , im just looking for some png's of ingame stuff so i can save myself the hassle of having to cut them off screenshots in photoshop xd
lavish gust
see pinned posts, someone already posted an archive with everything extracted
pseudo ocean
that first pin? i think its from 2018 tho
jade hawk
Is there some way to retrieve data from unique stash tabs? /character-window/get-stash-items does not seem to return anything :/
lavish gust
AFAIK, no
jade hawk
sad because im looking at ssf which means its not indexed either 😦
yes I do
yes it is, but it isnt an api page is it?
I was hoping I could use that unique ID but there seems to be no api for retrieving that data
yah im also toying around with what happens on that page, maybe theres something I can work with
im looking through it but for now its just magically appearing lol
yeah im also getting a lot of floating point random values
hehe
weird is they dont show the xp anywhere
scripts at bottom of pages yikes
ah I see it
Feels bad to have to run through every page to scrape the data :/
if it changes every time it is updated ingame im giving up lol
but I think its just a unique ID so probably not
ok it is persistent (or so it seems)
pseudo ocean
is it just me or is currency.poe.trade outdated af
i really dont wanna have to switch all my links to official trade thats gonna be so much work
pseudo ocean
does anyone know if the string behind a search link of the official trade website stay the same across leagues, i did a few tests and the string stays the same if you change league but will that be the case still in the new league ? wouldnt wanna swap all my links now for nothing
lavish gust
is the string a somehow encoded query or is it some random unique ID?
pseudo ocean
encoded query
i think, it stayed the same when i switch the league but kept the item selected
it was really easy with poe.trade because the items there have a unique recognizeable number but on the official trade its random letters and numbers
https://i.imgur.com/pYb1R5w.png here this for example
lavish gust
WTF the quality of this image
pseudo ocean
it highlighted the wrong part because i got a search going but at the end there of the .trade link you see want 4 have 820 , in that case 4 is chaos and 820 is the item
the link at the top is the official trade with the random string
oh sorry for bad quality, shouldnt be
lavish gust
this isn't even a jpeg, what the hell have you done?
pseudo ocean
its a png
lavish gust
png is lossless
pseudo ocean
anyway, since i swapped the league in the top link and the random string stayed the same i assume that its just the id for the item aswell so i can safely use the links or am i wrong?
sorry i meant i swapped the league on official trade and it stayed the same
lavish gust
fun fact: poe.ninja is already using such links
pseudo ocean
i would love being able to call the .ninja api somehow but im using wpf and i literally have no idea what im doing so i spent all morning/afternoon googling that but i gave up on that one ^
lavish gust
what you use for the application logic and rendering should be irrelevant
pseudo ocean
i was even taking a look at the currency cop sourcecode but hes using like 5 different .js classes and i just couldnt path through that with no knowledge
or rather understand the flow
lavish gust
I'm downloading data from poe ninja
and it's basically an array of links given to libcurl
pseudo ocean
i couldnt figure out a simple thing like getting data from an item output into a textbox, and even if i had it would be a whole other thing to integrate it into what i already worked on
lavish gust
So in short, general/beginner programming problems? You seem to have more trouble with the framework than PoE APIs.
pseudo ocean
yeah, i never learned it, started with this because its something i personally used and thought i learn something ^ basically google on every step
lavish gust
well, I started most of my projects this way
pseudo ocean
yeah thats what i heard people do, but its limiting
lavish gust
I don't code by trial-and-error. If I want to pick something big, I would actually spend some time learning it the dedicated way. Many things can be very frustrating by trying to learn them through trial-and-error.
pseudo ocean
well see, im gonna finish updating these links and functions and then take another look at api's afterwards, i wanted to atleast get it functional again before leaguestart, i kinda havent touched it since blight 😄
lavish gust
you working on some trade/stash management tool?
pseudo ocean
ah yeah its for trade
lavish gust
so ... another trade macro? Or something unique?
pseudo ocean
no its not a trade macro, its literally just a calculator
worthy cape
@pseudo ocean For a low-effort extract of a GGPK, use PyPoE. If you're extracting DDS files, make sure to enable the decompression option in settings to un-brotli them. If you're looking for UI images, many of them are in texture atlases indexed by UIImages text files.
(you can extract whole directories)
pseudo ocean
i just grabbed the pics from the wiki, it wasnt a clean or fast process thats why i thought might be easier to get them from the game directly but its fine 😛 thanks tho
violet path
@pseudo ocean can do a lot of stuff if you find a way to just use poe.ninja's api, can maybe help you. big time saver
lusty musk
can an item have more than one subcategory?
rapid pagoda
hmm, well, this is fun :3
wrote a basic GGPK file browser (no content viewing yet) in, like, an hour using https://github.com/lxn/walk
rapid pagoda
which one, the public stash API or the private one
-- assuming public given that question
edgy jacinthBOT
TakeRep <User:User> [Num:Whole number]
Invalid arguments provided: User "assuming" not found
lusty musk
public stash api
compact isle
just a heads up that live searches won't return any results until we catch up to the head of the river again
lusty musk
@compact isle could you clarify what verified exactly means in the public stash-api? its existence really confuses me
because if an item were gone, wouldn't it just... leave the item out of the stash?
rapid pagoda
In this context, I'm not sure it means anything. It's analogous to the "verified" tag on items in the forum
compact isle
it means it hasn't perform the verification steps that linked items on the forums do
so can be ignored for anything that doesn't relate to those
lusty musk
that doesn't seem particularly informative
alright
(I mean the attribute, not your explanation)
lusty musk
@compact isle can an item have multiple subcategories? it's an 'array of strings' but so far have only seen single string responses
compact isle
yes
compact isle
a warstaff is both a staff and a warstaff
lusty musk
I see
when I was consuming the 'API' from the website it was much easier to get examples for corner cases
now I'm struggling a bit
in various examples
like, synthesis items for example
is it an influence? I don't know
or does it add a synthesized attribute
or maybe synthesizedMods field
or maybe only the icon URL
compact isle
you can just check responses of items from your character window to see different properties
or find them on the trade site (although trade site is currently reindexing)
compact isle
^
lusty musk
ah
compact isle
should be
lusty musk
sorry for bothering you so much, just have a lot of questions haha
worthy cape
There are no dumb questions, just dumb .... ehm 😛
compact isle
this is the reason why good documentation should exist, it's my own fault really 😄
unfortunately a lot of my work for this league around that is being pushed back so we can finalise some legal requirements
lusty musk
'implicit', 'explicit', 'enchant', 'crafted', 'veiled', 'fractured', 'utility' is this list of mod types that can exist as <type>Mods field complete?
worthy cape
Legal requirements? 3.12 will be Path of Exile: Objection!
rapid pagoda
Templar Courts got nothing on this
lusty musk
inb4 fishingMods
rapid pagoda
fishing mods are in the normal mods list
lusty musk
alright I think I'm safe to ignore cosmeticMods assuming that's just MTX
rapid pagoda
oop, did I say too much? 🐟
compact isle
yeah
lusty musk
banne
@compact isle already found a disparity, to be fair it is in the extended section, but the site does not give category/subcategory info
compact isle
that's only provided in the public stash API yeah, I had a post in the website api forums about it
extended stuff from the public stash API won't exist in character-window etc
lusty musk
@compact isle by the way, what is the actual proper way to handle a stash of items result from the API, is it simply correct (ignoring efficiency) to 1. delete all previously known item ids associated with this stash 2. insert all provided items into database and associate with this stash ?
compact isle
yep
lusty musk
okay
that works if you don't care about finding out which items are 'new'
you have to do something more advanced if you do
compact isle
yeah internally for our trade website we do a diff over the different item sets to get new / updated / removed
lusty musk
do you simply trust the item_id? or do you do some sort of hash over the actual stats of the item?
IIRC in the past it's been really unclear if crafting an item in a stash tab updates the item_id
compact isle
we have item_id -> hash stored then iterate and compare
lusty musk
ah so it is a hash over the actual items properties/stats
compact isle
yeah and we don't really mind if the item id changes since we just consider that a delete / new
lusty musk
in other words, we can expect changes to this system in 3.11?
compact isle
nah I mean incrementally checking item differences works well but when the league ends all stashes become unpublic'd
lusty musk
ahhh
compact isle
you have to delete a lot of items :(
lusty musk
well for my purposes I'm not planning on deleting anything ever
worthy cape
What happens on SSF migration of an account, is that akin to league-end?
compact isle
yeah your tabs are unpublic'd
lusty musk
just marking items as not currently listed instead
compact isle
but a single account doing an SSF migration will be lost to the wind
rapid pagoda
... surely you can't list items for trade in a SSF league
cosmic saffron
you can set notes to whatever you want
compact isle
compared to ~80% (or whatever it is) of stored items being deleted instantly
lusty musk
you can publicly list them
lusty musk
@compact isle in case the API struggles (which it usually does at league starts)
is there some retry header it gives?
compact isle
struggles in what way?
lusty musk
right now I implemented quadratic backoff (sleep 1 sec, 2 sec, 3 sec, etc)
I don't know, getting 40X or 50X errors
compact isle
generally the only thing it does is be slow to return responses
lusty musk
alright
back when I was consuming the website API
especialyl with websockets
I had like three layers of exception handling and retrying
because there were so many points of failure
(fetches erroring out, websockets dying, the entire site being down, my own internet going down, etc)
compact isle
yeah for sure handling websocket errors isn't great
especially when in browser contexts you can't get the http code back
let me tell people they're being 429'd please >.>
worthy cape
Some day I'll stop mixing 418 and 429 up.
lusty musk
@compact isle is there an api for checking who's online or is the ladder the only way?
compact isle
there is but it's private and we don't really want to add new people to it
lusty musk
alright
lusty musk
@compact isle is the website reading through the river all the way from the start of the history or something? there's still 0 synthesis items listed whatsoever
compact isle
yes
lusty musk
I see
is that actually feasible? I assumed I had to get a recent id from poe.ninja or be stuck updating for weeks
compact isle
it is for us, the internal network is fast
lusty musk
do you have any estimate as to how many calls it would take?
then I can calculate given an optimal 2 requests/sec it would take me
I just naturally assumed weeks
compact isle
my math says ~18 million
lusty musk
18 million stashes or 18 million get-stash-item pages?
compact isle
it may be a 1/5th of that, I don't remember how the packet limit works for these
pages
wait no that's super wrong
lusty musk
18 million pages is 104 days of requesting pages in optimal condition under API limits
compact isle
yeah I copied the wrong number to our statistics so I'm not sure how many stashes are currently available
I'll let you know once we catch up (should take a day or two)
lusty musk
how many are you requesting per second?
if you estimate a day or two
then I can estimate your_rate/2 days or twice that
compact isle
that's historically how long it's taken, we do get slowed down by blocking while we process (fetching isn't async)
currently looks like ~2500 stashes per second
grave wren
Any chance you're keeping ascendancy and general buildtype info somewhere @fickle yew ? Would be cool to check old leagues %ascendany and %bow/mines/traps... Distribution
fickle yew
@grave wren Nope. I do have time machine ready for Harvest launch though 🙂
rain pasture
here's a really sexy placeholder image of indigo oil for you guys 😉
lavish gust
this is the new oil?
rain pasture
the name of it at least~
carmine merlin
I actually will need to make a placeholder image until I can get the official one from the website on launch
grave wren
rain pasture
is that just tinted violet or is that the real one?
grave wren
it's recolored to indigo
rain pasture
both look like purple to me 😛
grave wren
rain pasture
I can't believe they thought it was a good idea to stick blurple between blue and purple
grave wren
laughs in ips display
rain pasture
I'll probably steel yours though, especially if local is using it~
carmine merlin
@grave wren ty 🙂
floral dawn
grave wren
aint nobody got time for color blind modes
cosmic saffron
it'll just have a different slosh when you click on it
rain pasture
indigo oil: shake vigorously
cosmic saffron
but yeah thanks for the official image tix
worthy cape
@rain pasture Didn't they talk about annotating the passive tree hovers with spelled out colour names?
rain pasture
yeah I believe bex commented they're adding something for the tree
worthy cape
I have to keep telling myself that there's a difference between the middle and the right image in the blown up picture there, and that's with the benefit of seeing all three at the same time.
rain pasture
honestly, depending on your monitor I don't doubt they look almost the same
but fortunately I have tooltips 😄
worthy cape
Have I mentioned that I hate colouring gear? I've had to focus so hard on determining what colours there are each click that I've actually seen that there's slight differences in decoration on the sockets themselves at high resolutions.
rain pasture
I have a big problem where the gear sometimes makes it appear like there's a link where there isn't one
worthy cape
Hrm, logging in now there doesn't seem to be anymore... could've sworn there were little notches in the past.
hazy fiber
the sockets definitely were different other than just colour, but that was very long ago, dont remeber how big of a difference there currently is but its not what it use to be
minor charm
Does anyone know how poe.ninja gets the character info ? like is it an api endpoint or scraping characters from the profile page?
velvet fog
@minor charm PathofExile API https://app.swaggerhub.com/apis/Chuanhsing/poe/1.0.0
minor charm
Thank you 🙂
Is there any difference between using api.pathofexile.com/ladders/:id and pathofexile.com/api/ladders/:id ?
the docs on the poe website are using the first one but the swagger doc is using the second one
Excellent. ty
worthy cape
Passive tree JSON, items, and all that Fun Stuff: https://www.pathofexile.com/forum/view-thread/2873743/page/1#p23265876
rapid pagoda
Infused Engineer's Orbs, various Scrolls, Winged Scarabs, and things other than Offerings to the Goddess. Interesting
remote yoke
I hate when I can't get the tree data to work out of the box lol
slate rover
Is it me or are the images in that zip the wrong version?
mortal bone
I haven't had a chance to look at it, but if they added an oil that image might be missing
rain pasture
oh, that looks like a bug actually
slate rover
The images in the zip with the JSON data are the exact same images as 3.10.0
rain pasture
unless position actually matters now for anointments :X
slate rover
yeah i just checked
mortal bone
Yeah, they are. Interesting. @compact isle are the images for 3.10 still valid for 3.11?
slate rover
They dont look valid, everything is fucked up in my tree now XD
mortal bone
We are also missing: JewelSocketActiveLegion, and JewelSocketActiveLegionAlt
worthy cape
@rain pasture « Oil costs for Notable passive skills can now be revealed by holding alt while hovering over the passive. »
mortal bone
unless I missed that at some point
worthy cape
I assume that includes the colour name.
rain pasture
there's also no IndigoOil for any of the anointments..
compact isle
oh hm
worthy cape
"Fixed a bug where, due to some changes in how terrain is generated, a number of hideout templates became invalidated. They should work now." <- guessing this is Robber's Trench and other ones that used actual terrain generation for things.
"Fixed several modifiers that didn't have names." <- does this affect tools or has it never shown outside of the game?
mortal bone
dang, I can't really generate the sheet. I am missing 52 images 😦
compact isle
something is wrong there... I just compared file sizes but I guess they didn't export right
I have no idea how the missing oil managed to export without asserting / crashing
rain pasture
does this also affect the oil being missing from the tree recipes?
rain pasture
thanks! 🙏
mortal bone
I am missing 52 images since they aren't live yet, but that will get you looking mostly correct
rugged pecan
how does one open a json file?
dull roost
you won't want to, you can put it into the tree folder for PoB though, not sure about Emmitt's program though
rugged pecan
sorry i can build and fix comps but file formatting ngl im clueless lol
dull roost
its no formatting
worthy cape
@mortal bone It was directly linked from the actual patch notes as something that an end-user might know what to do with 😄
golden bane
JSON is just a structured text format, you'll need some kind of software to interpret the data in it like PoB
mortal bone
@worthy cape oh...
rugged pecan
ok so ill just delete it then
worthy cape
Due to the volume of the changes to the passive skill tree, we won't be listing each change out. Instead, you can check out the new tree for yourself here.
mortal bone
ah, I see
dull roost
I forget where pob stores the tree data tbh, anyone able to help there?
glossy flame
can anyone convert it to the lua format that PoB uses?
dull roost
isn't it in appdata or something?
glossy flame
C:\ProgramData
remote yoke
You don't need to convert it. If you throw it in as a tree.txt file PoB auto-converts it to .lua.
glossy flame
oh thanks
golden bane
@dull roost Default location is C:\ProgramData\Path of Building Community\TreeData
wraith glacier
Where and what do you put from the ZIP to get it to auto-convert? Never done it before
dull roost
@rugged pecan go to that location, create a folder called 3_11, and dump the skill tree zip into there
golden bane
Just wait a bit and you'll get your PoB update
remote yoke
The tree data isn't really perfectly working atm, so it's not really a great time to do it buuuut
dull roost
you gotta pull it out of the directory structure, cause PoB doesn't like folders in there
mortal bone
Yeah, you can download the one I have above and those spritesheets are the correct size at least
I will make a compare of the tree a bit later
I would like to add support for that in the application, but it is a lot of work sadly
rain pasture
there's a fixed zip coming~
mortal bone
Yep
remote yoke
If you're desperate for PoB to work, you can drop the Path of Building folder into ProgramData and it should work as well as intended, with minor sprite bugs still. Just don't update PoB until the official update is out or you'll overwrite it with an old version.
dull roost
if you do that DO NOT update PoB when it says update ready, it'll just revert it back to before you patched it like this
lol the witch area is a fucking mess rn
rain pasture
big fan of how you can anoint Shocking Strikes for free 🙃
broken cloud
What's the difference between these again?
Show EnchantmentPassiveNode == "Damage while you have a Herald"
Show EnchantmentPassiveNode = "Damage while you have a Herald"
tiny burrow
good lord that's a lot of changes
graceful frigate
@remote yoke may the Wraeclast be nice to you! Thanks a lot for that data
tiny burrow
Patchumz thing worked for me in the fork just fine
whole cloud
I took it, put it in C:\ProgramData\Path of Building\TreeData\3_11
started pob
and no change
bronze fable
dito
remote yoke
Updated PoB tree, unzip and drop the Path of Building folder into your ProgramData and overwrite whatever it says. Don't update the app until the proper update comes out.
Fixed sprites now.
Still broken Indigo sprite.
Which is a PoB problem, I believe.
Maybe?
tiny burrow
most likely
mortal bone
The indigo sprite was shared but Tix above if you need it
whole cloud
that worked, cheers
mortal bone
worthy cape
Any decade now.
versed ginkgo
How can I import passive tree things into PoB manualy?
gaunt tundra
scroll up and read
rain pasture
I love the 50 different reddit posts of people "fixing" the skill tree data one node at a time 😛
worthy cape
Too many cooks.
rain pasture
Tooooooooo many cooks
mortal bone
It would be nice to have an API for this kind of stuff, so everyone could just update their apps to pull down the prerlease data haha
ocean raft
GGG spending resources improving the public APIs. Ha! Good one.
mortal bone
to be fair, it doesn't have to be from ggg
we could just host a git repo for this kind of thing
I have a good chunk of the skill trees going back to 0.9.6
ocean raft
True true. Honestly the biggest thing missing is an API to retrieve item data, skills, mods etc. I once upon a Time tried building a web app for creating/sharing builds but gave up because of the difficulty of getting all the item data. I know RePoE exists but you still have to do a lot of work to get in a usable format.
And then every patch they change items and you start all over
velvet fog
but the passive skill trees format changed at 3.10
mortal bone
inland kestrel
Yo @mortal bone, do you also own http://poeskilltree.com or is that someone else?
inland kestrel
Thanks @velvet fog!
mortal bone
Yep, that is me. I haven't updated that for 3.10 sadly, so I don't know how soon I can get it up an running 😦
inland kestrel
I just took a quick look and it's not as trivial as dropping in the 3110_PssiveskillTree.zip contents somewhere
mortal bone
The format changed a ton in 3.10. Basically all the Opts.json data moved to the skill tree, so I have to work that into my parsing
inland kestrel
I love the comparison feature (between versions) and would find it exceptionally useful tonight. 🙂
mortal bone
Yeah, I will see what I can do
inland kestrel
I hear ya. Don't break yourself... I'm sure you have more exciting things to be doing 🙂
(but if you doooo... please @mention me :D)
mortal bone
Haha I don't have too much going on, so I will see what I can do
I want to switch the website over to blazor wasm at some point, so I don't have to write two parsers
frigid nova
@mortal bone your website is my favorite way to view diffs on passive trees!
Does poeskilltree generate similar diffs?
lusty musk
@compact isle is it safe to assume that a stash will not be given twice in the same public-stash-tabs page?
mortal bone
@frigid nova sadly, no. I would like to add that at some point
compact isle
yes
lusty musk
sweet
saves me a headache
@compact isle in general is a page of results internally consistent?
or could an item show up in one stash as well as another
compact isle
shouldn't be possible, the data is from the same transaction
worthy cape
Ugh, PoE Overlay went and became Overwolf-based while I looked away. Time to figure out another way to get a /hideout hotkey 😄
violet path
there's a community fork of poe overlay, non-overwolf
grave wren
Awakened trade macro is nicer in the usability imo so lots of choices
lusty musk
@worthy cape just maintain your own .ahk
worthy cape
I was happy to not have to mess with AHK ^^, and I don't mind vaguely realistic price checks
floral dawn
😂
worthy cape
Patch note distancing is important in these trying times.
grave wren
I just use the Firefox reader mode
Has a dark mode
Sepia as well if that's your jam
cosmic saffron
does user css work with reader mode?
carmine merlin
@compact isle https://www.reddit.com/r/pathofexile/comments/haiirv/bug_or_real/fv325qq/
Bex is saying that's its an interpretation problem on our end but the missing line "Nearby Enemies are Blinded" isn't anywhere in the JSON file. Have any idea if its an exporting error or intentional change?
grave wren
@cosmic saffron never tried tbh
lusty musk
that databases course did help after all I guess
async def get_new_items(self, item_ids, hashes, prices):
return await self.conn.fetch("""
SELECT q.id
FROM items
RIGHT JOIN (
SELECT * FROM unnest($1::text[], $2::text[], $3::real[]) x(id, hash, price)
) AS q
ON items.id = q.id
WHERE
items.id IS NULL OR
(removed = True AND last_change < now() - interval '5 minutes') OR
items.item_hash <> q.hash OR
abs(items.price - q.price) > 0.01;
""", item_ids, hashes, prices)
@worthy cape sorry I can't see what that emoji is lol
worthy cape
Sipping a cup of coffee.
oak apex
Hi, I followed a guide on reddit to update my POB but I still cannot see 3.11 passive tree. Is that because I am using fork vision?
hazy fiber
not the right place to ask but the fork version is capable of displaying new tree, not sure on method you speak of
oak apex
Sorry. Where should I visit if I have question in relation to POB?
carmine merlin
@oak apex I'm about to release an update to the fork version with the updated tree
oak apex
@oak apex I'm about to release an update to the fork version with the updated tree
@carmine merlin OMG
This is the talented man behind POB fork. Respect+1
fallow zinc
LocalIdentity, would you mind if I pm you a feature request (problem statement + possible solution)? I'd rather not make an issue, my only GH account is my work account. I can do it another time too, you have more pressing matters right now
worthy cape
@oak apex There's an army of little gnomes hacking on it.
carmine merlin
@fallow zinc yeah sure
fallow zinc
thank you
carmine merlin
3.11 tree PoB Community Fork update is out now
rain pasture
Oh, I guess I'll post my silly update, too. 😛 If you guys figure out the new ring anoints please hmu~ https://www.reddit.com/r/pathofexile/comments/haif8x/blight_helper_updated_for_311/
lusty musk
@carmine merlin is it possible to add an FPS cap to PoB? whenever I use it on my laptop the fans start spinning like crazy it's so intensive
hazy fiber
@lusty musk no as all the lowlevel stuff is handeled by simplegraphic.dll which we cant modify
lavish gust
you actually can, it's source code was published on one issue
but openarl did not really make it easy to build it
north hearth
anyone here have experience running the LIBGGPK app for parsing the content.ggpk? cant seem to get it t run
lavish gust
No, but have you got any error message?
north hearth
MainWindow.xaml.cs(1122,17): error CS0103: The name 'TextBoxOutput' does not exist in the current context [C:\Users\alsmi\VsCode\libggpk\VisualGGPK\VisualGGPK.csproj]
basically a bunch of those for different names
lavish gust
is this runtime or compile time error?
worthy cape
Reliably building .NET programs over time is "fun".
north hearth
bear with me as I have 0 c# expereince...I have downloaded the proj and .NET sdk and used dotnet run
what is the compile command?
worthy cape
What's your ultimate goal, btw?
north hearth
need to view content.ggpk to help parse data for PoE-Overlay-Community-Fork harvest update
lavish gust
@worthy cape well I hate string-matching UI configs, these sooner or later end up in problems like this due to misalignment unless there is a really robust CI/CD behind
worthy cape
PyPoE tends to be the go-to tool there, with pypoe_ui
north hearth
can i export the data with that?
worthy cape
It can export files.
north hearth
ok UI is a little rough on that one but I do have it running
worthy cape
😄
worthy cape
Seems legit.
grave wren
it's a bit on the heavier side
so discord said no
discord.errors.HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body In embed.fields.5.value: Must be 1024 or fewer in length.
😄
pseudo ocean
cheers m8
ah shit im gonna have to hold on putting that in anyway dont i
they are not gonna be updating the trade side with it before leaguestart :S
mortal bone
progress....
grave wren
Nah the small one is official
The bigger one is from me
Iirc tix is secret agent ggg
rain pasture
guess you did a pretty good job then 😛
grave wren
That media minor in uni sure paid out 
rain pasture
Day 2412: Received a compliment from an internet stranger. This crippling student loan debt is finally paying out!
grave wren
laughs in germany
my crippling masters cost was something like 1.3k€ combined
and half of it is from the first semester still costing 550€
mortal bone
annnnnnnd done...just need to upload to web site
mortal bone
@inland kestrel @frigid nova http://poeskilltree.com/?v=3.11.0-pre&c=3.10.0
that took a lot more work than it should have haha
inland kestrel
Really appreciate this. Awesome job Emmit!
mortal bone
Yeah, no problem. Thanks
shrewd ridge
What are the highlighted nodes? Nodes that were changed or are new?
mortal bone
Green = New, Yellow = Changed, Red = Removed
frigid nova
@mortal bone I believe nodes like Soul of Steel have changed, but arent reflected in this diff. Any ideas about that?
mortal bone
I am grabbing lunch, but I can check after. If the stats on the nodes changed it should show up
Oh, I see. The # of mods changed. I can fix that
also the recipe changed on a lot of these, but I don't show that currently. I can add that in the diff once I show them
uneven wren
getting this one on the tree diff on chrome, tree goes black
it does load fine initially, the error pops up after a while
mortal bone
I mean, the error message tells you what happened haha
I am going to assume you are using chrome? there might be something up with it
simple ravine
the browser has ADD?
cosmic saffron
yeah compare tree ate a lot of ram while pinch/zooming
mortal bone
Yeah it is going to
frigid nova
Also @mortal bone would it be possible to highlight with a different color when nodes move on the tree?
(Something like champion of the causee which moved significantly but had no change in mods)
mortal bone
@frigid nova Um, the last time I did that people were super confused. The main problem is, I can't have a good way of visually displaying something moved.
frigid nova
Right, perhaps just the color around it, without any indication to where it originally was?
I mainly would like to see it so I know what nodes to check with intuitive/meek/thread/unnatural
I can see how that would be confusing though!
mortal bone
@frigid nova alright, soul of steel is now highlighted. I will try and come up with something for moved nodes. That one might be a bit tough
frigid nova
Tons of thanks and appreciation for your hustle. This diff really is so crucial with giant passive tree changes
mortal bone
Updated compare image
torpid reef
Could somebody tell me if there's a faster way to look for my stash tab than (seemingly) endlessly requesting with the next_change_id?
(Public Stash API question)
fickle tusk
fickle tusk
number that indicates which stash tab you want the items for, as you only ever get the items for one stash tab with one request with 0 being the leftmost (first) stash tab afaik
mortal bone
@frigid nova something temporary, but it should work for the time being.
mortal bone
alright, that change is live. When you hover over a node it will show a pink highlight of where it used to be.
mortal bone
worthy cape
@edgy pilot The graphics/platform framework is indeed still binary, but it's the uninteresting part of it and you could reimplement the C API if you really cared to.
@rain pasture Go go oil time - https://www.pathofexile.com/forum/view-thread/2874150
rain pasture
beep boop
throws away the entire first useless table
that indigo map mod is pretty danged beefy
rain pasture
@worthy cape update'd thx ❤️
worthy cape
Got any particular goal in mind? Porting to different platforms, turning it web-based, or just debian-grade GNU/Freedom?
Most people tend to be fine with the occasional blob as it's not integral and all the things that matter are on the Lua side of things.
Feel free to spend your time on things, but please don't crusade.
mortal bone
The only issue with the project I linked, the ui framework is written with wpf on .net core 3, so it isn't fully Linux compatible. We don't have any plans that I know of to change that
Yeah, sorry haha the main thing driving the calculations is the engine project in the same team, so I don't think that project would be too hard to make work.
north hearth
does anyone know what data tends to be missing from the early content.ggpk torrent? Trying to decide if it's worth staying up late to parse data for a app or just wait for league start?
worthy cape
@north hearth Everything in /Data to start with.
north hearth
got it well I will have to wait till league start then
worthy cape
I wonder if my poediff still compiles.
@golden bane 20/20 gems out - does that help PoB?
@north hearth I ran a diff on 3.10 preload and "3.10.0" - https://gist.github.com/zao/5fc91bfa7995060fd97b54cc39b8a23c
Looks to have been mostly MTX tweaks and the addition of Data, from what I can tell.
The list may be a bit weird in places, as it's comparing standalone preload vs. shipped Steam.
velvet fog
PoB need full game data
north hearth
i need all the item stats from the data/ part though 😭
worthy cape
@velvet fog I’m running out of disk on the NAS for some reason 😄
velvet fog
you downloaded all version of poe? I extracted only Data and stat_descriptions.txt
worthy cape
I have 168 of the manifests atm, need to sort out delta compression for long-term storage.
velvet fog
using brotli ?
worthy cape
My last experiment used rdiff to get patch files and tried bzip2 and zstd on them, got a decent ratio.
Just need to write some scripts around it, and figure out what versions to derive from which.
rain pasture
I like how the zip of the POB fork installer is bigger than the exe
worthy cape
God-dammit this NAS is cursed... I can't run freshly compiled executables off it 😄
golden bane
@worthy cape Since we can't extrapolate for skill gem levels/quality other than 20, we usually wait until the release patch including the data directory in available. Also exporting from the GGPK is much easier than manually creating skill gems
golden bane
@rain pasture The size difference comes from the included Visual C++ runtime. The installer contains it in compressed form (executable installer), while the standalone contains it as uncompressed DLL. The installer itself only adds an overhead of 30-40 KB or so
@worthy cape Yes, LI might even stream the process this time
pseudo ocean
its been a while but i finally updated again, last time was in blight 
https://www.reddit.com/r/pathofexile/comments/hb84og/flipping_calc_20_release/
lavish gust
In recent filter information thread GGG gave following examples:
Show EnchantmentPassiveNode "Damage while you have a Herald" "Projectile Damage"
Show EnchantmentPassiveNode == "Damage while you have a Herald"
Show EnchantmentPassiveNode = "Damage while you have a Herald"
Is there any documentation what does = mean for string-based conditions? I know that nothing means "one of specified values, substring matching" and == means "one of specified values with strict 1:1 string matching". But what does =?
Also, I know that HasInfluence condition behaves differently with ==: If nothing is not present, it will match an item with at least one of specified influences. If == is present, it will only match items with all specified influences. This inconsistency isn't helping me guess what's the meaning of = .
lavish gust
Just tried some items with HasExplicitMod, there is no difference between = and (nothing) regardless how many mods item has / how many mods are on the line in the filter. Is this intended to have the same behavior?
lavish gust
nvm, just got a response from Rhys
lusty musk
@pseudo ocean man you really need some better contrast in that flipper thing
some minimal UI design
not black on dark gray text
NUM_REG = re.compile(r"((?:(?<![0-9])-)?[0-9]+(?:\.[0-9]+)?)")
what a regex to find numbers in modifiers and replace them by # haha
the negative lookbehind is necessary to distinguish -9% to Fire Resistance as a negative number from Adds 3-10 as two numbers
and yes that exists: https://www.pathofexile.com/trade/search/Standard/aL5KkdRSe
pseudo ocean
@lusty musk reads fine for me, which colour would you suggest
lusty musk
something with higher contrast
I mean I'm not going to use it, it's your tool, just suggesting you have bad contrast
pseudo ocean
Alright , thanks for the input, that never occurred to me, will take a look when I'm up :P
inland kestrel
is there any way to update my filter in game without having to go into the options menu and hit the refresh icon 😢
lavish gust
AFAIK, no
inland kestrel
Kinda wish there was an unbound-by-default hotkey to update it 😦
lavish gust
Auto-reload whne file is rewritten? I kinda want that thing too
inland kestrel
that would also do the trick
rain jacinth
@inland kestrel you could go to character select and back into game
inland kestrel
That is an alternative but I'm looking for something fast and non-intrusive
rain pasture
that's a lotta pub/subs 👀
pseudo ocean
@lusty musk https://i.imgur.com/uv5XPHu.png cheers m8 
lusty musk
^^
cosmic saffron
any plans to add seeds whenever they become flippable
pseudo ocean
technically i could add whatever the trade website lists on their "bulk exchange" site, so the currency trade, well have to wait and see if they get added and then also if its worth flipping them, my next plan would have been fragments and scarabs but i would not have made that in time for harvest, so im gonna get on that once my team is done playing together @cosmic saffron
cosmic saffron
yeah fair enough, I noticed that in window8 you also don't have indigos
pseudo ocean
because they also are not listed yet so i could not have added the links
i also wanted to do conqueror exalts but they are also not listed on the currency for some reason
mortal bone
Nightstalkers tooltip is a tad long... lol
lusty musk
so the first prototype of my indexer ran overnight, logging all errors....
and there were none
I'm so fucking skeptical haha
worthy cape
The good old programmer spidey sense when nothing goes visibly wrong.
near fossil
(don't know where to ask this, so if it's wrong, i will move it) Does anyone know if the Openarl Path of Building is discontinued? It still hasn't updated for the new league
rain pasture
I'm amazed that there's folks trying to vilify people for making a community fork of open source software...
simple ravine
Who is villifying?
eternal sand
That's hilarious if true. 'Fork open source software?' Who'd have thunk it? 😮
rain pasture
I'm not gonna name and shame, just dingdongs on reddit 😛
unreal ridge
guys pob of the community fork dont have new notables or keystones ?
rain pasture
it does
unreal ridge
im not seeing that one than regenerte rage instead of mana
simple ravine
Oh reddit. Who would've guessed.
frigid nova
@near fossil It has been updated at the beginning of the last few leagues with passive tree and gems - although I have no information about the current one. Check out the Community Fork for a more actively updated version.
@unreal ridge The fork is in the process of gaining support for new passives, keystones, and items - expect gradual updates within the coming weeks.
@unreal ridge Also the specific keystone you referenced is a legion jewel keystone, iirc, which the fork doesn't support
rain pasture
the passive tree for sure has been updated, though not everything applies properly to damage yet (e.g. exert and stuff)
whoa I got a skill point
narrow eagle
Anyone have any stats on public_stash_tab api processing times? I know poe.watch had some but that seems to have been down.
fickle tusk
You can periodically check https://poe.ninja/stats to get a rough idea @narrow eagle
narrow eagle
Yeah I guess thats fair
simple ravine
what kind of processing time?
narrow eagle
Json deserialization, and if people are storing that into a DB how long it takes to process into the DB
mortal bone
You can hand the blob off to another process to finish handling it
You really only need the first part of the request to get the next id and have another thread process the rest of the request
simple ravine
I have an urge to try to do this with Azure Durable Functions
narrow eagle
Yeah thats how I was doing it at first. However I was adding the stashes into a queue to process into my DB. Problem is what do I do if I have over 11k stashes in my queue from two days ago?
Do I cap the queue at like 10k stashes process them all and start handling ID's again?
mortal bone
Oh geeze. That is a lot lol
simple ravine
that means you need to beef up ur processing hehe
narrow eagle
yeah
mortal bone
Yeah, you need to pump those numbers up
narrow eagle
I mean, it takes less than 1 second to deserialize and move to structs
simple ravine
so I guess indexing is a problem for u?
narrow eagle
problem is it then takes anywhere from 9s to 15s to process the full ID into the DB
simple ravine
relational or nosql?
narrow eagle
relational, postgresql
simple ravine
ah, the one I am not so familiar with
but I do have a vague recollection of pgsql have some funky sharding stuff nowadays
where it'll automatically balance the cluster?
knotty patio
which one is the updated fork for POB?
simple ravine
@knotty patio #1┃endgame-help
mortal bone
Doesn't postgress offer an option now for storing sql blobs?
mortal bone
Neither...
simple ravine
@narrow eagle single node?
simple ravine
potentially look at index fragmentation. what kind of PK do u have? many relational constraints?
narrow eagle
Otherwise Id want to do some fun stuff like caching ect
Theres only like 2 relational constraints. Problem is the item table is like 32 columns with over 500k items
simple ravine
Well, depending on your usecase, you want a source of truth up-to-date either way
narrow eagle
yeah thats fair
simple ravine
500K is not much though
mortal bone
That is nothing lol
narrow eagle
Fair enough
simple ravine
You'll have 100s of millions when you're done
mortal bone
I would just assume the laptop is slow tbh
We had a table in prod that had 1.7billion rows and no performance issues
simple ravine
I'd take a second look at the PK, FKs and any other indexes
those tend to be what slow your inserts down
I inserted 1.5 billion rows into Azure Synapse (cheapest pricing tier) in less than 10 minutes
narrow eagle
If i floated out a 3k transactions per second number, idk how that is relative to other stuff
simple ravine
transactions are kind of arbitrary without context 🙂
narrow eagle
ah, thats totally fair
mortal bone
I just find it weird it would take so long to process the id
simple ravine
what's your primary key? do you have it sorted?
narrow eagle
Uhhh let me check
simple ravine
my hunch is leaning towards PK/index
narrow eagle
Sooooo, right now I am using the item ID as the primary key
I also had some questions about when that changes in regards to moving characters ect
simple ravine
Ok, pgsql has some kind of explain command to give u the definition I guess?
narrow eagle
I have my migrations
simple ravine
id changes when moving owner, and sometimes in other scenarios too
like when u chaos it(?) etc
narrow eagle
is there a way to spoiler large text?
abyss_jewel bool,
additional_properties item_requirement[],
art_file_name text,
category text not null,
corrupted bool,
cosmetic_mods text[],
crafted_mods text[],
descr_text text,
dubplicated bool,
elder bool,
enchant_mods text[],
explicit_mods text[],
extended item_extended_data not null,
flavour_text text[],
fractured bool,
fractured_mods text[],
frame_type frame_type not null,
h int4 not null,
icon text not null,
id text primary key,
identified bool not null,
ilvl int4 not null,
implicit_mods text[],
influences influence_type,
inventory_id text,
is_relic bool,
league text not null,
locked_to_character bool,
max_stack_size int4,
name text not null,
next_level_requirements item_requirement[],
note text,
properties item_requirement[],
prophecy_diff_text text,
prophecy_text text,
requirements item_requirement[],
sec_descr_text text,
shaper bool,
sockets socket_type[],
stack_size int4,
stash_tab_id text references stash_tabs(id) not null,
support bool,
talisman_tier int4,
type_line text not null,
utility_mods text[],
verified bool not null,
w int4 not null,
x int4 not null,
y int4 not null
)```ok, so I'm no pgsql expert, but I am imagining it working similarily to mssql and mysql
where text = varchar(max)
that's gonna make your indexing hurt like a mofo
mortal bone
Oh, yeah, you need to optimize that table a bit
mortal bone
I think so
simple ravine
example Id
3c8f1dc99dabab14f888755f133dfb532dee09edb2058dd7138d4409c6923f72
narrow eagle
that makes sense. It's stuff like descriptions ect that I was iffy about but for an id I should probably figure out the max size
simple ravine
looks more like a sha of some sort
64 in length.
so make it a char(64) (fixed length)
non-unicode, simple ascii or whatever
should speed up inserts dramatically
mortal bone
Since you have data you can query to find the max/average lengths you have and optimize your table a lot
narrow eagle
Alright, well that gives me some stuff to do. Thanks!
simple ravine
could also drop any FKs and manage that programatically, will increase a little headache, but improve speed
wonder if those ids are deterministic
Wonder how many stash fetches you'd do every month
probably around 1 million
would be interesting to see if I could do this cheaply with azure functions
rain pasture
Azure cost estimation is terrifying
simple ravine
Depends how you look at it 🙂
rain pasture
we've started using it at work and I find myself jumping between several different pages to try to figure out the total cost for a project
rain pasture
I always think I just need one thing, but then I read into it some more and end up checking the price of X and Y individually
simple ravine
Good that you're starting to adopt cloud though 👍
rain pasture
this certainly looks to bet a better approach
it would be, if it didn't take our IT department 18 years to get new components hooked into the internal network
simple ravine
I'm overseeing a project which is globally distributed (3 regions right now, more to come), which require strict GDPR and HIPAA requirement compliancy.
You could say this project's not gonna be cheap heh
you need VPN/Express Route because your InfoSec department is ancient, yes?
rain pasture
I'm still a fan of the monolith VM approach. I like just having a big sandbox. But obviously it can't scale like something like Azure Functions would
simple ravine
I've been "cloud native" for many years now, not looking back heh
rain pasture
Hah, I have no clue. I let them handle all that mess. But yeah we're super ancient
simple ravine
but I do enjoy a bit more of the "distributed system" chunky projects
rain pasture
"cloud" was a scary word to most of the execs until maybe two years ago
simple ravine
I'm still having some customers who believe their datacenter is more secure and better than cloud.
I've stopped arguing though. I can't be bothered trying to convert these people.
rapid pagoda
@narrow eagle tbh, the first question I'd have is what parts of the item data you actually care about for your project. A data structure which stores everything you get back from the stash tab API will be hugely wasteful and very difficult to analyze
for example, properties like flavour text and image URLs are entirely redundant to knowing what base type (or unique) it is
lusty musk
@narrow eagle
narrow eagle
:?
rapid pagoda
you will.
rapid pagoda
I just did
rapid pagoda
having mods exist as a separate table is going to cause you some pretty serious grief once you start trying to query the database
lusty musk
@rapid pagoda do you have any experience with relational databases?
rapid pagoda
@lusty musk Yes.
rapid pagoda
Consider what a query like "items with flat phys and % phys damage" would look like
lusty musk
yes, that's two joins?
rapid pagoda
And what does the query plan look like?
lusty musk
EXPLAIN SELECT id
FROM items
JOIN has_mod as m1
ON m1.item = items.internal_id
JOIN has_mod as m2
ON m2.item = items.internal_id
WHERE m1.mod = 1 AND m1.modtype = 'explicit'
AND m2.mod = 2 AND m2.modtype = 'explicit'
AND m1.average > 3 AND m2.average > 4;
" Workers Planned: 1"
" -> Nested Loop (cost=42.65..12087.99 rows=12 width=65)"
" -> Nested Loop (cost=42.22..11717.17 rows=81 width=8)"
" -> Parallel Bitmap Heap Scan on has_mod m2 (cost=41.79..5030.60 rows=795 width=4)"
" Recheck Cond: ((modtype = 'explicit'::modtype_t) AND (mod = 2) AND (average > '4'::double precision))"
" -> Bitmap Index Scan on has_mod_modtype_mod_avg (cost=0.00..41.45 rows=1351 width=0)"
" Index Cond: ((modtype = 'explicit'::modtype_t) AND (mod = 2) AND (average > '4'::double precision))"
" -> Index Scan using has_mod_pkey on has_mod m1 (cost=0.43..8.40 rows=1 width=4)"
" Index Cond: ((item = m2.item) AND (modtype = 'explicit'::modtype_t) AND (mod = 1))"
" Filter: (average > '3'::double precision)"
" -> Index Scan using items_internal_id on items (cost=0.43..4.58 rows=1 width=69)"
" Index Cond: (internal_id = m1.item)"```all indexed
rapid pagoda
"Index scan" does not mean "indexed"
I'm more familiar with mysql's terminology than psql, but I'm pretty sure that's saying it's going to take all modifiers with a qualified value of mod 1, then scan every mod on those items
gaunt tundra
with all due respect that schema is bonkers
gaunt tundra
uh orlp
lusty musk
I really don't see an alternative to it
rapid pagoda
orlp's, but yours would have similar issues -- it's inherent to the overall design of item has-many mod
lusty musk
either way I won't be querying based on stats
narrow eagle
You have any recommendations @rapid pagoda? Always looking to improve
rapid pagoda
@narrow eagle a document database, tbh
narrow eagle
And just store the item as a blob?
gaunt tundra
the index scans on the keys there are fine
the bitmap scan is likely cause of the row count and that there isnt anything else taxing the db
narrow eagle
I guess I'm not super familiar with nosql and document databases, so I am not sure how storing an item works
Since you don't necessarily have columns but only keys and values?
gaunt tundra
actually orlp, you got any other query examples?
just joining on has mod would be fast ftmp if this db is read only
n+1 is somewhat worryin
lusty musk
n+1?
gaunt tundra
just adding more joins for more lookups
mortal bone
Postgres actually can do a document-style db in a way you might be more familiar with. There is a column type jsonb that allows you to store the json blobs for stats and things
gaunt tundra
^ it's wicked fast too
lusty musk
there's no way that's going to be faster than my approach
mortal bone
There is a ton of features for getting data out of them as well
lusty musk
EXPLAIN SELECT id
FROM items
JOIN has_mod as m1
ON m1.item = items.internal_id
JOIN has_mod as m2
ON m2.item = items.internal_id
JOIN has_mod as m3
ON m3.item = items.internal_id
WHERE m1.mod = 1 AND m1.modtype = 'explicit'
AND m2.mod = 2 AND m2.modtype = 'explicit'
AND m3.mod = 124 AND m3.modtype = 'implicit'
AND m1.average > 3 AND m2.average > 4 AND m3.average < 2
LIMIT 10;
@gaunt tundra
"Limit (cost=1.86..306.90 rows=1 width=65)"
" -> Nested Loop (cost=1.86..306.90 rows=1 width=65)"
" -> Nested Loop (cost=1.43..304.77 rows=1 width=77)"
" -> Nested Loop (cost=1.00..296.51 rows=1 width=8)"
" -> Index Scan using has_mod_modtype_mod_avg on has_mod m3 (cost=0.56..76.29 rows=26 width=4)"
" Index Cond: ((modtype = 'implicit'::modtype_t) AND (mod = 124) AND (average < '2'::double precision))"
" -> Index Scan using has_mod_pkey on has_mod m2 (cost=0.43..8.46 rows=1 width=4)"
" Index Cond: ((item = m3.item) AND (modtype = 'explicit'::modtype_t) AND (mod = 2))"
" Filter: (average > '4'::double precision)"
" -> Index Scan using items_internal_id on items (cost=0.43..8.26 rows=1 width=69)"
" Index Cond: (internal_id = m2.item)"
" -> Index Scan using has_mod_pkey on has_mod m1 (cost=0.43..2.12 rows=1 width=4)"
" Index Cond: ((item = items.internal_id) AND (modtype = 'explicit'::modtype_t) AND (mod = 1))"
" Filter: (average > '3'::double precision)"
Huh, that's cool. Anyone have a recommendation for document db?
rapid pagoda
elasticsearch.
rapid pagoda
What is your goal, then?
lusty musk
I care about fast updates and compact storage format
I store items for machine learning
rapid pagoda
Then store each item as a JSONB blob
lusty musk
it's too big
and since I will have to process it into tabular format anyway might as well do that before inserting into a database
mortal bone
This goes over the differences between postgres jsonb and nosql: https://blog.couchbase.com/postgres-jsonb-and-nosql/
Gives a nice overview of what it does and how to use it
narrow eagle
Oh, thats exactly what I was looking for
And it seems that diesel has support for jsonb out of the box
👀
gaunt tundra
@lusty musk scrolling up i dont even see that u posted any issues-- carry on, dunno where the critique of that schema came from 😄
lusty musk
@gaunt tundra could you rephrase that?
I don't understand what you mean by me not posting issues
gaunt tundra
rephrased: i dont know why im even looking at this, the only thing u did was post a schema
lusty musk
also I really don't see the issue with my scheme, people, document databases aren't magic
they still need indices to do fast lookups
gaunt tundra
im not gonna make up use cases in my head if u dont have any issues lol
lusty musk
and I don't know what indices they could use I couldn't also add
rapid pagoda
@lusty musk tbh, if you're just storing a bunch of items for later analysis I'm not sure you need a database at all. what it sounds like you're really after is some way to ETL the incoming stash tab river into a more compact form that's more directly useful for your ML tasks -- am I correct?
lusty musk
mostly
I need to keep track of which items are listed right now
and which items are new listings and which are old
I want to be able to slice my data and only extract, say, all rare rings posted in the last 5 days on Harvest softcore
gaunt tundra
is all this local only
lusty musk
yes
lusty musk
it does though, keeping track of this efficiently requires essentially a database
rapid pagoda
right, so given those requirements a first step I'd take is storing the volatile data (item id, listing date, removal date) separately from the static data (item id and all the item properties)
lusty musk
or something hacked that would just be a database but worse
gaunt tundra
is it slow now
lusty musk
no
gaunt tundra
then youre fine?
gaunt tundra
hence what i said earlier...
rain pasture
rip stash typ
if you're keeping this relational, I'd suggest splitting items into shared and unique values
e.g. there's no need to have the name, icon, width, height, etc. duplicated for every stash entry
so one table would have that info, and another would be the stash entry with the x, y, price, etc.
lusty musk
huh
how is that duplicated?
each item has exactly one name
so why would they be in separate tables?
rain pasture
but your items also have x, y, price, etc.
lusty musk
I deduplicate icons because the urls are terribly long and not that many distinct values exist
yes, each item has an individual x place?
and an individual price?
that's not shared at all
rain pasture
so if I have 10 starforges in my stash tab, I'll have 10 things repeating the starforge name, icon, width, height, etc.
just with different x/y or prices
lusty musk
for uniques, sure
rain pasture
I mean, for any item
lusty musk
but eh
rain pasture
the item meta data is repeated for every entry in the stash
lusty musk
not x, y though
w, h sure
also icon isn't true
you can have different icons for the 'same' unique
because of different influence for example
or mirrored
rapid pagoda
Those are postprocessing effects. The image is the same
rain pasture
that's getting into a totally separate issue 😛
but we can pick easier columns for examples
name, base, width, height, max stack size, these categories maybe, not sure what those are
lusty musk
wrong
max stack size depends on the tab as well
can be higher if it's in a different tab
rain pasture
sure, keep that in this table then 🙂
lusty musk
name again is assuming uniques
and you're assuming that legacy items don't exist
in general you just keep adding assumptions that just don't hold in general
PoE has some weird items you come across when you scan over everything
trust me
rain pasture
sure, you clearly know the data better than me. but I don't doubt there's some amount of duplication going on here
lusty musk
there's a bit but not much
rain pasture
just thought I'd mention it if you're going to have millions of rows in this table or whatnot
rapid pagoda
Perhaps a useful way of thinking of it is, the data you get in the stash tab feed (or wherever) is a heavily filtered version of what exists in GGG's game server database
Cross-referencing against game data files, esp. to parse out properties which are provided through baseitemtype or itemvisualidentity, will help you cull out a lot of the redundant properties
golden bane
Sounds like you don't want a relational database then, if you don't want to normalise it
rain pasture
can I get a shoutout for Boyce-Codd Normal Form?
golden bane
4NF should be attainable in many cases
rain pasture
I just love the name tbh
lusty musk
my point isn't that I don't want to normalize, I'm just pointing out that many of the 'shared' relationships and 'duplicated data'... just aren't
golden bane
My 2c: Either you get your DB scheme in at least BCNF or you might be better off with a nosql approach
rapid pagoda
Oh shit, it's up. Starting download now 🙂
also I had some bad ideas and decided to add a river parser to pogo 😄
Does anyone have a full schema for that? I've found a few fields which don't appear to be documented anywhere (like cisRaceReward)
rapid pagoda
https://github.com/oriath-net/pogo -- a suite of command-line tools I'm working on
rapid pagoda
right, I've seen that one as well, I'm wondering if there's any authoritative source for the schema
keen hollow
Is the tree JSON data in the GGPK?
lusty musk
we already have the tree json data
unless you want to confirm no last minute changes happened?
rapid pagoda
@keen hollow not that I'm aware of, I think it's probably in PassiveSkills.dat, PassiveSkillBuffs.dat, etc
keen hollow
we have incorrect tree json data
rapid pagoda
do you mean for 3.10 or 3.11?
keen hollow
for 3.11
rapid pagoda
Well, I don't think we have Data for 3.11 yet either, so 🤷♂️
keen hollow
Right, I guess what I'm asking is if we can extract the correct tree data from the GGPK once the patch goes live, or if we have to wait for GGG to release a fixed json file for us.
rapid pagoda
You can get the passive skill tree JSON off the web site when the patch goes live. It's used as part of the passive tree viewer
keen hollow
I see, ok
rapid pagoda
View source on https://www.pathofexile.com/fullscreen-passive-skill-tree, scroll to the bottom, and you'll see the tail end of the passive tree JSON
keen hollow
got it, ty
sweet copper
Can anyone help me with this i have uninstalled 3 times and i still get it?
i also try updating but there are no new updates
south gorge
Does anybody know how to get "extended" data from Stash API?
https://www.pathofexile.com/forum/view-thread/2784742/page/1#p22948552
I can't see this node in API responses.
velvet fog
only public stash api have extended data
south gorge
@velvet fog Thanks, so the stash need to be public...
broken cloud
@south gorge Are you using get-stash-items? The extended property is not present there
south gorge
@broken cloud Yes, I'm using it, where it is if not there?
rapid pagoda
No, the extended property only shows up in the stash tab river (/api/public-stash-tabs), not in requests to the private stash API (/character-window/get-stash-items)
south gorge
Thanks for help guys, it seems that I will have to still determine item categories based on item name. I hope some day they will add this property to private stash API.
broken cloud
@south gorge extended is also returned in responses from the trade site, so you can get it by searching for items sold by you (this seems a bit wasteful though)
rapid pagoda
The private stash API is kind of an unintended API. It was only meant to be used on the web site, not by third parties
south gorge
@broken cloud It would require users of my tool to have premium stashes that are public.
@rapid pagoda Did they specify somewhere that it is prohibited to use private stash API in third parties? If so I will need to change it to public.
rapid pagoda
It's not prohibited, but it's not the use case they had in mind when they built it out, and there's some discomfort about some of the methods that third-party apps use to get access to that API (like copying session cookies or entering username/password into apps).
south gorge
@rapid pagoda Thank you :)
rapid pagoda
unrelated to current conversation, but here's the schema I ended up with for the stash tab river -- please let me know if I've missed some obscure field https://github.com/oriath-net/pogo/blob/master/river/river.go
fickle tusk
@sweet copper it seems you are trying to run it diretly out of the zip. unpack the zip first into a seperate folder and then try to run it. If thats not the case or it still does not work create an issue with some info at github depending on what version you are using
worthy cape
Which one of you fine people is 500:ing all of github?
pale wind
How are y'all getting around the vulkan borderless windowed overlay issues?
I just tried a few scripts and things other people said worked for them and haven't had any promising results yet
worthy cape
I'm fine with being minimized by tools, biggest annoyance is mashing Alt-Enter a few times to get it fullscreen again.
Beats the race-the-6s-disconnect-timer of old ^^
austere heron
༼ つ ◕_◕ ༽つ LOCALIDENTITY TAKE MY ENERGY ༼ つ ◕_◕ ༽つ
pale wind
hmm nvm it's actually working fine now
setting the resolution 1px wider than my screen stopped it from reverting to fullscreen mode
rain pasture
@keen hollow did you catch v2 of the tree data?
pseudo ocean
Will GGG ever give us a few options of different coloured mouse pointers in-game the current one gets lost too easily
keen hollow
@rain pasture Do you have a link just in case?
rain pasture
If you downloaded it right when they posted, you got the original bugged version. They Ninja fixed it to this.
celest coral
Is the full map tier data available somewhere already or only the base map tiers are known from the itemfilter info?
worthy cape
The first tier a map occurs as is available on the forum.
keen hollow
@rain pasture Ok, it looks like the one I already had. It's still missing the AOE blind for Borni n the Shadows and doesn't have the Combat Control notable
rain pasture
Ah okay, was hoping I could fix it for you 😁
worthy cape
(which I guess is what you refer to by the itemfilter info)
worthy cape
Fun fact - Steam has rate limits <_<
Logging 'hunter2' into Steam3...Unable to login to Steam3: RateLimitExceeded
celest coral
(which I guess is what you refer to by the itemfilter info)
@worthy cape yeah, that's what I've meant, what I need is the complete data for tiers at all awakener levels though
fickle tusk
everybody not rate limiting public apis today is living dangerously
rain pasture
Wow zao I like how your login is all stars
rain pasture
so it sounds like the GGPK doesn't have the sweet deets most folks need for their updates?
worthy cape
Never does with the preload
It’s tradition to go full panic as patching goes live ^^
rain pasture
haha gotcha
fickle tusk
this just feeds into the usual developer work flashbacks where you have to deploy to prod on friday evening and find a last minute bug you have to fix in two hours
velvet fog
Audio/Dialogue/NPC/DaughterOfSinPet ... a tencent only sin pet with 37 chinese audios
rain pasture
hey chuanh am I crazy or does poedb have the tower anointment descriptions hiding somewhere?
rain pasture
ahh there it is in the 2nd link, thanks
velvet fog
just rewrite the site with new tab system
rain pasture
I'll be keeping an eye on this for the new ring anoints 😄
since they didn't publish the descriptions
deft robin
Question, is there any tool out there similar to mercury trade where it helps with incoming trades?
umbral mortar
So, I was using looty addon and couldn't get it to work, turns out that having an expired username returns 404s from poe and looty won't move on
opal temple
pob isnt update anymore ? or it just take time this league ?
because i cant add a lot of new build to look at them, seems a lots of people use alternative pob fork, but i dont wanna mess around with my files for now
worthy cape
@opal temple There's nothing manual to edit if you run the community fork from its installer, the manifest editing was a thing of old when converting an existing installation.
OpenArl's PoB updates when Arl gets around to it, which may not be the timescale you'd like.
Community PoB has new tree and will have things that become known at launch shortly after launch.
If you want minimal intrusion, you can unpack a standalone installation of PoB - that won't touch anything outside of its location.
Tool developers can't add things that don't exist yet ^^
opal temple
i didnt get anything but ok
fickle tusk
@opal temple TLDR: original pob is updated at a slower pace than the community fork. if you want to stay up to date as soon as possible get the zip version from here https://github.com/PathOfBuildingCommunity/PathOfBuilding/releases and unpack, nothing on your current install will be changed
opal temple
ok so its just a second POB, i'll look for it thanks
rain pasture
but i dont wanna mess around with my files for now
it picks up on all your existing builds
grave wren
@worthy cape is this you
worthy cape
I settled for xdelta3 as patch algorithm in the end, takes 20-some minutes to generate a delta for the hefty packs.
bsdiff is too memory-hungry, rdiff I terminated after 3.5h of churn.
velvet fog
patch server down
velvet fog
on pc too
worthy cape
1.69 GB for preloaded torrent
velvet fog
ya, for Data/
young osprey
its 7 on standalone without torrent
pseudo ocean
i had 5.5 on standalone with torrent
velvet fog
1.69g with torrent
worthy cape
internet goes brrrr
pseudo ocean
well i also deleted my old content.ggpk before pasting the one from the torrent so that might be the difference
wait now its loading another 7
sum ting wong
rapid pagoda
New data files:
/Data/DamageHitEffects.dat- ||
/Data/DeliriumStashTabLayout.dat|| - a bunch of Harvest-related files
/Data/IndexableSupportGems.dat/Data/LabyrinthBonusItems.dat- ||
/Data/MetamorphosisStashTabLayout.dat|| /Data/ModEquivalencies.dat/Data/Rulesets.dat/Data/ScarabTypes.dat/Data/SessionQuestFlags.dat/Data/TriggerBeam.dat
pearl sinew
where can i get the preload torrentr
worthy cape
@pearl sinew News posts, but there's little point now.
Regular patching is faster if you've got the standalone already.
pearl sinew
yea i do
worthy cape
(torrent is 21.9 GiB + 1.69 GiB fixup-patch)
pseudo ocean
https://i.imgur.com/1lvlivv.png did it really get 12 gb bigger over night
i had the torrent ggpk then it downloaded 5 in the launcher then i restarted the game and now its loading another 7
worthy cape
You're holding it wrong 🙂
pseudo ocean
worthy cape
@merry wraith My 3.10.2d was 23.4 GiB, torrent-patched is 24.4 GiB.
(steam is a chilll 35.9 GiB now, compared to 32-some before)
worthy cape
^^
pearl sinew
ha ha i just got intoo 18 min
pseudo ocean
im just pissed off i got the torrent downloaded for a reason and now these additional slow ass 12 gb made me be late to queue
worthy cape
Historically they've advised to not try to patch with it until a particular point in time, I wonder if you hit just the wrong spot?
mortal bone
they tell you that though
worthy cape
Maybe something with ProgramData and being installed into Program Files?
Not sure how the elevation situation works with the patcher, I always install elsewhere.
mortal bone
the torrent doesn't contain the game data and it also doesn't allow for upgrading. You will, on average, have less of a download if you just wait
pseudo ocean
they said 3 hours before, i waited that long
dull laurel
whats the infrastructure behind filterblade? Does it need a server/api? I thought you only build your stuff in the browser/singlepageapp.
frigid nova
Who here working on PyPoE spec?
lost widget
anyone know where PyPoE gets its expected row sizes? I've tried padding stable.py with extra fields to get rid of a rowsize mismatch error but that doesn't work
rapid pagoda
Expected row size can be determined by searching for the BBBBBBBB (🐝🐝🐝?) separator in the raw data and dividing its position by the row count (plug: pogo analyze will do this for you). What file are you having trouble with?
lost widget
baseitemtypes.dat, expected 203 vs real size 211
trying to figure how to tell pypoe to expect an extra 8 bytes
woeful sphinx
add 8 bytes worth of fields to the spec at the end
rapid pagoda
Looks like some Harvest-related data got added to the end. Try adding
('SomeHarvestThing', Field(name = 'SomeHarvestThing', type = 'ulong'))
to the end of the spec
woeful sphinx
disclaimer: that will get you another error, they seem to have removed at least one field in BaseItemTypes somewhere
rapid pagoda
Are you sure? I'm looking at the two side by side and the fields generally look the same otherwise
woeful sphinx
maybe it's caused by something else, but it can't dereference Identify_AchievementItemsKeys, it tries to offset far beyond the file length
lost widget
i must be crazy because that's exactly what i tried and it's still expecting 203
('Padding1', Field(
name='Padding1',
type='int',
)),
('Padding2', Field(
name='Padding2',
type='int',
)),
Are you sure you're adding the padding to fields and not virtual_fields?
lost widget
thanks
rapid pagoda
@woeful sphinx That sounds like AchievementItems changed. Hmm, no, that's not it.
woeful sphinx
it fails before going to another dat file
grizzled seal
@hushed relic attempting to auto-sort divination cards in filterblade throws an error (poe.ninja returning no JSON data), I can send it to you if you want
woeful sphinx
now that's weird: it says "BaseItemTypes.dat": Specification row size 203 vs real size 211. that should mean I'm 8 bytes short, but padding with a single ulong/two int fields it still says 207 vs 211. if I do one ulong and one int, I get the aforementioned unpack error. if I do three int fields, the file seems fine and it complains about another one.
fickle yew
No economy for you!
hushed relic
@fickle yew lets just shut down both our sites for a day, see what happens? :3
sounds like a good 1st april fools joke
grizzled seal
panic