tooldev-general

[modest path]

it's totally matching the control-alt-c text as separate array items per line

"--------" matches everything

[pallid wraith]

Probably just the gRAnts

[worthy cape]

Best thing about all this is all the words you don't think about that ruin your day.

[pallid wraith]

There is no way to actually get handed actual currency exchange rates.
The only api is the stash river, which is basically just every public stash change that happens, which poe ninja bases their data on. If you want to not spend an inordinate amount of time and processing power and such, you'll just want to use PoE Ninja's API to get an idea about rates for Items on the trade website.

It also doesn't include the Currency Exchange feature from in-game, which I have no idea if it's what you were asking about, since you also said Trade Site, while they're two separate systems.

[pallid wraith]

Indeed

So it's best if you just make searches to look for matching combinations if possible

So you at worst just have to manually read an extra item sometimes

Or fix up the search

Rather than risk missing out on some good rolls because your search wasn't perfect

[worthy cape]

For interactive use it's probably fine. More fun for peeps that use regexen to filter bulk buys.

[pallid wraith]

Indeed

Because those usually need all the space they can get

[worthy cape]

640k 50 characters ought to be enough for anybody.

[pallid wraith]

And on the topic of the limit.
I don't think it's a technical limitation that made them put it so low

[modest path]

Given that if you backtrack-ddos it it's just going to break your client, yeah

[pallid wraith]

I'd rather guess it's there to stop the players from overoptimising looking at items via it

[modest path]

literally no reason to not have it larger

*technical reason

[pallid wraith]

In the sense of, we want players to actually read their items

Not just be able to chuck in an omega sized regex to only see the perfect rolls

But also, possibly to avoid players trying to look for too specific things

[worthy cape]

Put in a full six-mod item and spam chaos until the frame lights up 😉

[pallid wraith]

Yeah

A healthy sprinkle of saving the players from themselves is always nice

[modest path]

hey zao, try "(.*.+.+.+.+.+.+.+aaa)" - note - this is malicious and will possibly crash your game

Yeah I'm pretty sure I crashed my game

I think .*.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+aa is about maximum on redos in the 50 char limit

that will absolutely crash the game again

[worthy cape]

Crash or time out?

[modest path]

I mean

I didn't wait all night

but the exponential increase in backtracking per .+ is pretty severe

and given that they're running it per line per item in your stash tab

[worthy cape]

Quadastrophic results.

[modest path]

it shouldn't eat all your ram, just all your cpu, so if you've got time maybe it'd just time out

by time out I mean eventually complete but with the server gone long ago

[frank drift]

it's possible they changed/upgraded libraries at some point

[modest path]

I think they're currently using the ecmascript mode with capturing disabled

maybe with some custom unescaping up front

oh, no, no fancy escaping

other than the space/quote shit which still makes no sense

anyway, what am I doing with my life, it's almost 2:00AM

[cyan void]

you can use \s in poe regex??

wat

isnt that for newlines

or am i stupid

[worthy cape]

Whitespace.

spaces, tabs, vtabs, etc.

[cyan void]

feel like thats also linebreaks

[worthy cape]

In multiline mode probably newlines and whatnot.

[cyan void]

thats how Inuse it normally

yeah ok

makes sense

was trying multiline the other day and didnt work

but if for whitespaces itll do

[clever thistle]

\n for line breaks(ish)

or \r\n because windoze

[oak estuary]

This would've been me, I'm so glad I was already asleep to avoid the nerd snipe

[modest path]

The item descriptions use \r\n but the regex matching appears to be separate matches against each line, not one match against a multiline string - so ^ and $ can match in numerous places

[modest path]

https://learn.microsoft.com/en-us/cpp/standard-library/regular-expressions-cpp?view=msvc-170 - It's in excemascript mode with grouping disabled , so you can use things like positive/negative lookahead/behind assertions, unicode escape sequences, and (very usefully) zero width word boundaries and negated word boundaries

so "blood\b" matches blood but not bloody, and "blood\B" matches bloody but not blood

[compact wigeon]

I imagine it works similarly to how the passive skills work. Stats are in an array of strings, one for each line.

Some stats have multiple lines, in which case they're one entry in the array with \r\n in the middle.

If so then the search is just iterating through the stat array and checking against each, plus the other stuff. Maybe.

[modest path]

The caveat is that the searched text is the full control-alt-c text as '-----' matches everything

Hmm, yeah, should test \n and \r\n, and \s+ on an item with a mod with \n's in it

oh also there's definitely a mod that has a literal \n in it (vs the string \n

A mod translation text string I mean

[compact wigeon]

The passive trees are enough headache for me, I'll leave item sleuthing to you

[modest path]

"\n" appears to match all maps

but \n doesn't lol

nice

oh "\n" matches all map stash tab tabs but no maps

Oh, any quoted string matches all map stash tab map sections

so that's just broken as shit

[pallid wraith]

On a side note, if you're searching the map tab, remember to click through all of them before trying anything if the map stash tab itself isn't one of your 4 leftmost tabs

[worthy cape]

[modest path]

This explains an experience I had long ago lol

[worthy cape]

The composite tabs are weird like that, too much data to preload so they tend to cache what you've seen.

[frank drift]

more like lazy load, they're not loaded at all unless they're one of the first few as above I think

the contents of the sub tabs I mean

[worthy cape]

I mean "preload" when you open the tab and they're not one of the focused inventories.

[frank drift]

are they cached though? I never noticed

[worthy cape]

I honestly don't remember.

[pallid wraith]

Well, more like, all tabs work like this, it's just that the map stash tab has tabs inside of itself. And also one or two other premium tab types
No tabs are automatically loaded in until you see them, unless they're leftmost 4. With the caveat that subtabs also get loaded in as well for leftmost 4.

I'm not sure if I'd say it's cached locally, because the server also doesn't know tabs exist if they aren't loaded in, which you'll primarily notice for the purposes of spending currency in benches or vendors

Although you could argue that that's just the client being the one to tell you if you can or can't buy something

[worthy cape]

I would lean towards it being the client that blocks you there, but we can't tell from here.

[pallid wraith]

Indeed

[brittle oasis]

the poe skill tree github gives the orbit radii as [0,82,162,335,493,662,846] ...but i dont get accurate positions unless I multiply the radius by .72 ....what am i missing

[worthy cape]

The web tree in general has a bit of a scale to it I think, so depending on what assets you use you may have a bit of a mismatch.

[lone ingot]

some progress https://gyazo.com/cf250a4ff829ad5430f1f645a88758c9

I need to get those icons fixed at some point lol

[lone ingot]

@worthy cape do you mind linking to where the wiki's art assembling logic is? 💚

[worthy cape]

Flasks and transfigured gems
Maps

There's a table of floating point constants somewhere in the file too to get the shades right for things.

I should probably have converted those to human-readable numbers after pulling them out of the render pipeline, but hey 😄

It's pretty much all in that item.py.

[lone ingot]

thank you

[worthy cape]

Take care around sRGB images so that your math is done in linear space, otherwise it'll be subtly or catastropically wrong. Ask me how I know 😛

[cyan void]

hihi just signed up on reddit to post my project

i have the biggest smile on my face right now

[worthy cape]

They seem to have misspelled "Nik Post".

[cyan void]

[brittle oasis]

For anyone who's familiar with the official skill tree export, what's the logic behind the nodes in and out arrays? Almost seems random at first glance but I'm sure that isn't the case

[worthy cape]

Probably just some artefact of how it's constructed.

[cyan void]

@velvet fog might be a long shot but do you by any chance have a json that holds all possible item prefixes/suffixes? similar to this glossary https://www.poewiki.net/wiki/List_of_item_modifiers

[hazy fiber]

iirc distance from the start, in goes towards the root, out goes away from root

its not actually useful because you need to traverse the graph properly due to how nodes are connected anyway, so you can just combine them into a single "connected" array for most purposes

[brittle oasis]

ty, thought I might be missing something

[mint karma]

Yep. Using a simple lookup table.

I've been meaning to get the optimizations of multiple mods in forever, but it's been so slow. The LuT is generated from around 15k tokens, trying to ensure that the regexs are exclusive from all those weird things they can match.

I've figured out a way to make the optimizations with a secondary lookup table. Currently a work in progress.

Gotten around 40 mods with combining mods down to around 10ms, but it's not perfect yet.

[clever thistle]

is the source code for the generator available somewhere?

[mint karma]

Not currently, but I can share some of the code if there is anything specific you're wondering about.

[clever thistle]

no problem, was just curious, seems like a difficult task 🙂

[mint karma]

It's very hard, especially with ensuring both it being correct and fast. Hopefully I'll have an uptime during this week. But IRL work takes most of my time

The good part is that I can do a lot of preprocessing using LuTs

[modest path]

There any evidence that the trade site doesn't use the exact same stat text rendering code the game does?

I'm looking to compile a large (complete?) number of test cases for mods -> displayed lines for the preformat function application and rounding code

[warm dirge]

Anyone here using python script to automate potion / flask use? Safe from the hammer? 😄

[hazy fiber]

I mean, its a bannable offence, so I wouldnt do it

[worthy cape]

I recommend accepting the constraints of the game.

[frank drift]

PoE has potions?

[warm dirge]

im a day2 player. LOL

kk thanks for the info.

[mint karma]

Finally happy with the speed of the regex map optimization! Now just for some more testing to ensure there are no false positives

Regex optimization:   0 ms, number of mods:  7, result:  6
Regex optimization:   2 ms, number of mods: 13, result: 11
Regex optimization:  12 ms, number of mods: 18, result: 15
Regex optimization:  33 ms, number of mods: 26, result: 20
Regex optimization:  22 ms, number of mods: 35, result: 26
Regex optimization:  38 ms, number of mods: 40, result: 31
Regex optimization: 100 ms, number of mods: 50, result: 35

[cyan void]

ayoooo

what you cooking

[cyan void]

i think i just did what you are doing rn like last week

https://exile.re

100ms for 50mods seems crazy efficient though

[mint karma]

Good job on that webpage! 😄

I've been working on and off on this optimization over the last year, never been happy with the result. But I am pretty happy about how it turned out now

[cyan void]

you have a page for it already?

[mint karma]

https://poe.re

[cyan void]

NO WAY

😭

this was my design inspiration

[mint karma]

hehe, I could tell! thats a big complement to get

[cyan void]

is your update going to make my site obsolete

only reason i made this was because your site wasnt grouping things it could

[mint karma]

There are a lot of areas where regex would be really nice to have, like magic items!

[mint karma]

yeah, I've been cooking it for a very long time

[cyan void]

im currently working on something for rolling items

its a pain

[mint karma]

I can imagine, how are you approching it?

[cyan void]

very complicated which is why its pain, mapping of each mod and the possible values and how quality influences them

then generate a regex to capture that number range plus something unique for the mod name

[mint karma]

ah yeah, if you limit it to magic items only it becomes a lot easier, because then you can use the affix names

[cyan void]

oh im not going after items at all

im going specific mods

for example this

id generate a regex for 42-45 (or whatever it would be with quality)

[mint karma]

ahh, you want to match both the range and the mod?

[cyan void]

and then a string unique to "to cold resistance"

yeah

coudnt think of any other way

would still fit 2-3 mods should suffice for rolling

[mint karma]

of the ice$ is unique for that mod and tier tho

[cyan void]

cant use regex on that text

atleast i cant

[mint karma]

oh, you can, if its a magic item 😄

[cyan void]

ahhhh

[mint karma]

so of the pupil

is the suffix, and dancer's is the prefix, in this case

[cyan void]

mhm but then you cant target specific mods

which i guess thats why its limited to magic items

hate limitations! we go the nasty way

[mint karma]

I would be big if you figure out how to do it for rare items!

[cyan void]

i mean my approach should work for that

just need to write some more code before i can see it work

i mean this is a bad example but would be something like this

the tier is represented by the number range

that would be t2 cold res for example

only issue i have is that i've only been playing for 5 weeks and I need to properly figure out how quality works and if theres cases where it behaves different

guess i focus more on this now and let you handle the maps

[modest path]

getting a dump of a bunch of the responses to the stash api firehose is harder than I thought it would be

poe only having full oauth for auth to the apis is annoying given I just want to tail it for a while as a one-off

the version used by the live search would be fine if it didn't rate limit iself by trying to be low latency instead of efficient

[lilac jungle]

Item quality shouldn't influence mod magnitude unless it's a tincture, or you are using certain catalysts on a ring

[pallid wraith]

Or amulet, or belt

[worthy cape]

"Jewellery"?

[pallid wraith]

Indeed

[wise sand]

I've been working on something to somewhat easily track kingsmarch shipments/rewards to maybe figure out some more of the exact mechanics (and because I was bored and had time). If anyone's interested, I might attempt to make it a little more portable than it is now and share it

At this point, at the press of a hotkey it can read the current shipments being send and the currency rewards received (if the appropriate uis are opened)

https://imgur.com/Yj9RxvT

If anyone is interested in the project or has suggestions for how to analyze the date that can be gathered this way, please do message me

[frank drift]

might want to ask Prohibited Library

[wise sand]

Thanks for the tip

[stoic locust]

still wishing trade site could do like... a Weighted Sum / Price sorting 😄

but i guess its enough to just put in a price max when doing a weighted sum search

[modest path]

There are a lot of things the trade site could do better

If I didn't hate javascript with a passion I'd have made a browser extension that makes it filter the mods dropdowns by mods that are actually compatible with the base item type you have selected

[wind garden]

@compact isle Is there a reason why live searches don't get triggered by a player logging on? Only if they're new or modified listings.

[compact isle]

because live-search is about items not about account changes

and tracking account changes would quickly break the entire system

[wind garden]

break? or just be too costly to implement

[compact isle]

a single player has a million items
they log in
a million items need to be instantly checked against all live search queries

you can see how this isn't really feasible

[wind garden]

ah, makes sense

thanks

[nocturne plaza]

@fickle yew hi rasmuskl, i wanted to get your permission regarding a wrapper kind of app i made based off poe.ninja, mainly for learning frontend. i sent you a friend request before remembering about checking here in this server.
is it okay if we have a chat or something?

[flat star]

Please tell me how I can send messages in the game and read chat messages? Like in PoeTrade

[worthy cape]

Chat entries are collected in logs/Client.txt in the game installation directory. It may omit messages from people on accounts that may by law not be logged, like Korean Kakao clients. You can monitor this file using whatever file system monitor functionality your OS/language provides and parse new entries.
Message sending is typically done by putting the message on the clipboard and sending inputs to open the chatbox, paste, send. The song-and-dance around it to get it robust is a bit icky, something like Awakened PoE Trade may be a good inspiration.
Note that you are still subject to the requirements outlined in the Third-Party Policy of the docs around actions and determinism.

[flat star]

Thank you very much for the answer! In my case, I want to send messages from the site. I wanted to make a "response" like on the trading site, so that when you click a button on the site, a message would be written in the game

[frank drift]

you cannot send messages like the site can

you could copy something to the clipboard and paste it in game

[worthy cape]

There's no infrastructure for interacting with the game session from the outside, the only mechanism there is is the trade site and its Direct Whisper, and that's purpose-build and cannot be used.

[flat star]

of course, in my case, an option is to make a browser extension that will connect the game process in the system with the page in the browser, but this is already somehow complicated

[frank drift]

you do not interact with the "game process" at all, the only thing you interact with is the clipboard

[worthy cape]

Well technically, you've got to be flinging inputs in the general direction to get it to do the thing once you've got the board populated.

Anyway, regardless of the mechanism, adhere to the rules and recognize that no tool is fully safe.

[flat star]

Tell me more, please, I saw that on the site hideoutshowcase.com there is a mark MTX. How can this be implemented?

[frank drift]

I have no idea what you're referring to

[frank drift]

parsing the game files, or using community tools that parse the game files

you could potentially also just scrape the store to get a list of what decorations are non-free

[worthy cape]

.hideout files reference decorations by an unique hash.
The game data files have reverse-engineerable table files that describe the decorations.
One might be able to relate those to MTX shop tables, either by some flag or by some manual work against the store.
Never looked at what was premium or not, had it on the plans for the Hideout Editor eventually.

[pallid wraith]

Though weren't there some mtx that you can't buy any more

Or is that just actual hideouts

[frank drift]

[flat star]

is there any information on where and how to access such a list from the site to identify an item from HO

[worthy cape]

The easiest resource for end developers for data access is RePoE, but it doesn't seem to track any hideout metadata: https://repoe-fork.github.io/

HideoutDoodads.dat64 has the list of doodads, the Id references BaseItemTypes.dat64 which has a hash, which is the value denoted in the exported .hideout files.
You need a whole bunch of tooling and schemas to access the table files.

Some amount of data mining competence needed.

[flat star]

I would like to understand this, since using these games will significantly increase my capabilities) Thanks for the help

[worthy cape]

You might get going a bit by doing exports from the tables mentioned with https://snosme.github.io/poe-dat-viewer/ to JSON and relate those files.

There's a ton of things to do and learn around this, I'm probably not the best person to get someone started on it these days.

[worthy cape]

@rapid pagoda Had a chat with someone about oriath.net and they noted that there's some peeps in there that they would like to see voicelines for - like Sanctum.
The last league mentioned explicitly seems to be Expedition, while there's Siege of the Atlas stuff there too.
Is the project a bit dormant or just an oversight?

[rapid pagoda]

It's kinda dormant I'm afraid. I've been focused on other things lately

[worthy cape]

Haven't we all? 🙂

Tell'em I appreciate their work, that site helped me plenty

[velvet fog]

check this https://poedb.tw/us/NPCs

[slim sorrel]

hello, does anyone know where Awakened Poe saves its settings, images, etc? Am trying to copy it from one PC to another

[slim sorrel]

Found it: it's the apt-data folder in AppData/Roaming/awakened-poe-trade path, FYI

[flat star]

Hello everyone! I already realized that in this chat I can find help from a wonderful community, so I have another question for you and a cry for help!)

I'm trying to set up OAuth on my site, but for some reason I'm running into a problem that my request is defined as GET no matter how hard I try. Maybe I'm doing something wrong or misunderstanding... The poeLogin method receives the code and redirects to /oauth/enter, where the poeEnter method is processed. But sending it returns an error {"error":"invalid_request","error_description":"The request method must be POST when creating a token"}

[lone ingot]

more progress https://gyazo.com/6835ee1872903e829ef2a034c66bd4ef

[jovial tartan]

I'm a paper and pen type of person who likes just keeping notes on what I have to do, so I decided to translate that into a web app. Still working on little things.

[velvet fog]

Path of Exile Spend Calculator:
https://github.com/DanielTaranger/poeTransactionCounter/blob/cf748b6bc716961636051278460284f08f4672a9/poeTransactionCounter.js

[frank drift]

I would rather not know

[worthy cape]

It's over nine thousaaaand.

[flat star]

I'm a paper and pen type of person who

[modest path]

ModDomains (the enum table), it seems like things referencing row 38, which is not in the community schema, are "things without mods" or something - e.g. stackable currency? Anything else known about that?

hmm, there's some cluster jewels that are id 38

like magic items

yeah several other kinds of magic items appear to point to id 38

[worthy cape]

Note that there's only 37 entries in moddomains.dat64.

[modest path]

yeah for sure, but there's references to 38

aaactually

I think we're hitting a bug

in our code

yeah for sure

[worthy cape]

[modest path]

hahahah, it's a bug around finding items by name, but items that don't have names can't be foudn that way

[worthy cape]

38 should be a perfectly cromulent value for a reference to moddomains.

But yeah, probably not quite as many sources as you had 😄

[modest path]

I'm pretty sure there are references to some enum tables that reference values that don't exist in the schema, but that isn't one of them necesarily

oh fuck me

magic items don't have a name, they have a base item type that has the prefix and suffix pre/postpended

so it's only one line in the item text

"Jade Flask of the Conger"

is a magic "Jade Flask" with an "of the Conger" suffix, - all on one line though

rares get the base item class alone, with a separate name

[worthy cape]

Approximately 84% of all baseitemtypes have a moddomains of 38.

Sneaky.

[modest path]

yeah for stackable currency and other shit it makes sense

for item items though, it's just a bug around how magic items are displayed that is breaking my code

By that I mean a bug in my code parsing how they are displayed

There are no bugs in GGGs decisions, they just are

I am so curious if they've changed anything about fundamental data formats in poe2 that will require a whole new round of RE work

[worthy cape]

All integers now middle-endian.

It's going to be interesting to see how much the community rallies up for EA and whatever other tech patches we'll get.

I won't be participating much at all apart from keeping my fundamental tools running I reckon.

It's kind of amazing that DAT(64) has lasted so long throughout the game's lifetime with only minor changes.

[modest path]

Are you familiar with blizzard's MPQ lol

[velvet fog]

Aren’t most games now converted to CASC?

[modest path]

Yeah but mpq still lasted liike

a long fucking time

[worthy cape]

Only from afar, seems like a lovely accessible format.

[modest path]

warcraft 1 to diablo 3 or some shit?

[worthy cape]

@modest path I forget, have I asked you what project you're working on?

[modest path]

@worthy cape high performance crafting simulator engine for ML crafting route discovery

optimized for cost/steps/whatever

I nerd sniped my math friend with a paper about using ML crafting on PoE

and he said "the hardest part is going to be getting the data"

We're still getting the data

@worthy cape There's a moderate amount of vaguely useful rust utility shit coming out of it, much of which overlaps with current js/python shit

[worthy cape]

[modest path]

e.g. our tables import creates vecs of structs with pointers to other table rows for references, so it's super performant

All the table loading rust is generated from the json schema with jsonnet

[worthy cape]

Self-referencing pointer soups, the thing that Rust shines at 😉

[modest path]

After the 65 Ukrainian men and women arrived at an Oklahoma Army base for Patriot training, they made a special request: to add more soup options to the menu.

“They like soup. They’re very, you know, soup-centric. So we added some soup.”

Rust is very soup-centric

[worthy cape]

My current DAT tooling is in glorious Pythong. It's not fast.

[modest path]

our dat tooling is still using the json export from poe-dat-viewer, but it just slurps it all at startup

I'd prefer to read the files directly but given it's working, we should clearly concentrate on things that aren't

working on control-c and control-alt-c item text parsing atm, so I can hopefully validate my stats description parsing/preformat function/interpolation

I was going to slurp a bunch of example items down from the public stash api but apparently my request to the oauth peeps wasn't good enough

so I just set up a thing to watch my clipboard and manually copied almost 1000 items from my stashes lol

[worthy cape]

Lost to the winds or rejected for some reason?

[modest path]

no response

[worthy cape]

Ah, that's good.

Well, not good, but you know, good.

[modest path]

Better than being told to piss into the wind

They're busy, I understand, I also have a job

Just trying not to give myself rsi and/or abuse the trade site api to get a dump of legit items though

[worthy cape]

Was gonna suggest maybe pulling something from ninja or pobbin, but those builds may be a bit tainted.

[modest path]

I /know/ someone around here has a database of that shit, but I haven't been able to find anyone

there's a rust project to tail the api and send it to a stream processing queue, so clearly they are storing them and processing

[worthy cape]

I believe PoB has/wants testing infrastructure where they pull a fistful of builds off the web to do some sort of sanity check.

[rapid pagoda]

the filename-dependent encryption thing in mpq is fairly ridiculous tbh

[worthy cape]

There was also that fella that had "trending/similar" builds in PoB, with the controversy and all.

[rapid pagoda]

(way back in the day I wrote a tool to semi-automatically analyze scripts in wc3 custom maps)

[modest path]

was there controversy other than 🤡

[worthy cape]

Players had way too little to do at the time and thought that the feature was both making the build creator universe a bit lopsided by featuring builds without the accompanying materials, as well as completely over-interpreting that the person that had the website/service also had a patreon for costs.

One of those things that makes you disappointed in players and makes you think "this is why we can't have nice things".

[frank drift]

the controversy was the website creator allowing content creators to pay to be featured

low quality and spammy

and this was put into Path of Building

[modest path]

gross

[worthy cape]

It's a thing that happened, in the past.

[modest path]

honestly sounds less bad than 🤡

but not god

good

[worthy cape]

Anyway, to your problem, there's some amount of builds out there that are probably using legitimate items to some degree; particuarly Ninja.

At least the ones on Ninja that aren't from the user-uploaded section.

[modest path]

yeah, all the items I'm currently testing against were manually copied by me out of the game, so all legit

plenty of bugs revealed by that lol

[worthy cape]

Not quite sure what I expected, but searching the PoB issues/prs for the word "builds" wasn't super narrowing.

[modest path]

hahahah

[lone ingot]

if anyone implements the current py/js tooling for parsing dat files into tables in rust let me know 😄

[worthy cape]

I had macros in the past to derive. Completely murdered compile times.
I think I moved to build.rs codegen instead before abandoning Rust.

[formal kraken]

I just have the tables I need handcrafted with only the columns I need https://github.com/Dav1dde/pobbin-assets/blob/master/src/dat/tables.rs

[frank drift]

artisanal

[modest path]

Ours is autogenerated but using the json - If there's a lib for parsing them generically, I can probably tool up the autogeneration to use that instead of the json, though I probably won't do that until there's pathofexile-dat style cdn file pulling to get the raw tables nicely

I can pop our data crate out into another repo to make public if people are interested in using/contributing

[worthy cape]

For data access I've built most of my tools around Inya these days, but still haven't made manifests ergonomic yet.

Most programs don't take kindly to downloading and processing 100-200 MiB of JSONL for a particular game version.

[modest path]

Not familiar with inya and it appears to be impossible to google?

[worthy cape]

Sounds about right.

[modest path]

it this about to be a deez nutz joke

[formal kraken]

inyaaaaaaa nutzzz

[worthy cape]

It's my personal datastore and "CDN" of all PoE versions since around 1.0.

Manifests for each release mapping bundled/loose files from path to content hash, and a big pile of content-addressable blobs.

All available over HTTPS or (sooooon GRPC).

Did you know that you can run out of ports on a client system if you open too many HTTP sessions? 😛

[modest path]

Our rust generation is currently Public Schema JSON -> jsonnet code -> rust lib encompassing enum types, and lazy-loaded vecs of Table Row structs, where reference fields are easily dereferencable pointers to other Table Row structs from the target tables

@worthy cape oooo

[worthy cape]

Ask me about my 500 GiB postgres database.

[modest path]

Yes, yes I did. But I had to set up a test rig to test a physical server accepting like 1.6M connections way in the past for work lol

Iirc ephemeral ports run out at 64k even if you have multiple IPs to source conns from, so you have to explicitly specify ports, instead of using the default of 0 - then the kernel's connection tracking uses src/dst ip/port tuple so you can't have more than 64k connections from/to the same ofthose, then linux has a hard cap of 1M filehandles per process so if you want more than a bit under that per linux box you have to use multiple separate processes

I think that service at peak had 28 million TCP connections?

"fun"

@worthy cape is that data all in postgres?

Also is it durable/backed up? Would be happy to keep a copy if there's a mechanism to do so

[worthy cape]

Yeah. Used to have it on the filesystem in directory shards, but BYTEA can carry a lot 😄

I haven't figured out a good way to store the manifests in a queryable manner, right now they're pre-cooked JSON blobs in a table.

[worthy cape]

Technically reconstructible from the ~15T of source material, currently lives on a single VM on mirrored HDDs.

[modest path]

15T is the raw content.ggpks or something?

[worthy cape]

Surrounded by the jankiest Pythong you could imagine to obtain and ingest new releases.

[modest path]

or I guess post-cdn thing

raw bundles

[worthy cape]

Yeah, original game trees as obtained from Steam so a mix of legacy GGPKs and bundle soups.

[modest path]

That is very interesting to me

I assume the distribution of content sizes is ggpk sized to absolutely tiny and in large quantity

[worthy cape]

Some day I'll figure out how to adapt it for PoE2 too.

[modest path]

well hopefully poe2 will only have cdn bundle shit lol

[worthy cape]

I store both the actual loose bundles (+videos/audio/etc.) as well as the bundled files within.

GGPKs I ingest and keep my pristine spare copies on an array at home.

[modest path]

Yeah, obviously the most future-proof storage mechanism, but goddamn inefficient given the duplication

[worthy cape]

There's 120 gigabytes of .bundle.bin.

Pulled all of them down recently to test a decompressor.

[modest path]

I'd probably personally try to get the decompression/unpacking close to known 100% not losing anything, then delete the originals. Alternately, try to strip the compression off the originals and store them still as one file but uncompressed, then use some sort of data store that can do partial file deduplication for files with some overlap or something

[worthy cape]

I keep originals so I can completely screw up the production system and know I can regenerate it over a few weeks of compute time.

[modest path]

Def get it

We do the same thing at work

[worthy cape]

I also wrote some deduplication code for GGPKs that cuts them up into skeletons without the free-space chaff and references to external deduplicated objects.

[modest path]

Nice

[worthy cape]

Technically lossy so I still have original GGPKs lying around; but I doubt there's much real archival purpose to having any orphaned files or garbage free-space around.

[modest path]

@worthy cape Can updates to bundles be differential or are they always compete replacements for a file previously in the content, where content is either a ggpk or an amalgam of bundles from the cdn as expected by the game for a version

[worthy cape]

One of the nice things about the fairly granular bundling is that most bundles are the same across releases.

Bundles that are changed between releases generally have no similarity from the outside as they're compressed.

[modest path]

but they're not like, used to intelligently update anything, they're just a full overwrite of any files they touch right

[worthy cape]

I'm not quite sure if I follow your line of thought.

Patching-wise, bundles are atomic files subject to Steam or Standalone patching logic.

[modest path]

So the bundles and CDN shit are just an abstraction for making downloading/updating differential right - So the eventual result of either a content.ggpk download, or a CDN bundle slurping process at a particular version, is to get the equivalent of a content.ggpk as far as files available in that content.ggpk goes

[worthy cape]

The GGPK is a client-side VFS for the Standalone client containing a number of files and some bookkeeping metadata.

[modest path]

Fo sho

And the CDN bundle shit is just an alternate way of populating the same VFS right

[worthy cape]

Steam has the same set of files, but loose on disk.

The GGPK contains bundles verbatim, they're not processed into something else when patching.

A bundle is a block-compressed set of concatenated file payloads, indexed by the index.

[modest path]

Ok so the differential portion is the sets of files in the bundles, the bundles are just a way of grouping them

So a bundle is just some arbitrary number of files from content.ggpk, grouped so they can be downloaded together without making a billion requests, then put in the VFS by overwriting files in content.ggpk with them

[worthy cape]

No.

The GGPK contains bundles as-is.

[modest path]

So the bundles are unbundled on read by the client?

[worthy cape]

There's an index mapping from logical file entry to bundle file and decompressed offset+size.

(index also has a compacted string table that can be used to iterate logical directories and locate paths)

[modest path]

So a CDN-style update causes additional bundles to be added to the GGPK directly? Does it GC entirely replaced bundles?

[worthy cape]

Yep. If you want to access a logical bundled file, you find out which bundle and where in that bundle it's located and decompress the blocks that the file spans across.

[worthy cape]

Yep.

[modest path]

Are bundle updates always associated with a version, or do they make changes without bumping versions too

[worthy cape]

On Steam, game client versions and game data updates are in decoupled depots.

You can have new client executables without new data, and vice versa.

There's no "version" per se in the data itself, only data that fits version X.

Given the data files for a particular game release, you can't tell what game version they're for.

My index works out the game version for a release by squinting real hard at the EXEs.

[modest path]

Are the bundles at least versioned with their own version or timestamp so you can tell what order they go in and what precedence the files in them have?

[worthy cape]

That's a bit of an odd question.

[modest path]

on brand

for me

[worthy cape]

There's no overlap in bundles.
Each bundled game file exists in exactly one bundle.

You won't find Art/UIImages1.txt in more than one bundle in a particular game release.
The index maps from the path to the exact bundle and where in that bundle to extract the payload.

[modest path]

Ooooo - got it

So files don't move between bundles, but an individual bundle can have a file added or removed from it or whatever

[worthy cape]

Each bundle is nothing more compressed data, organized in individually decompressable blocks resulting in 256 KiB of uncompressed data.

[modest path]

But if you update a bundle it's a full new version of every file in it, even if only one changed?

hence bundles tend to be smal

l

[worthy cape]

The separation of files into bundles is arbitrary and can change over time; done as a whole.

[modest path]

But if a file "moved", for it to be aviailable to the game client the same way, both bundles would have to be updated at the same time

[worthy cape]

If you keep the same set of files in a bundle and modify one of them for a release, the whole bundle is remade.

[modest path]

Yeah

Got it

[worthy cape]

Recently there was a significant reorganization of bundles into a flatter tree of bundles for directories, but in general they tend to be fairly stable.

[modest path]

I was expecting something more like bigtable works where there's a stack of versions for every file with a newest, with the older ones being garbage collected out, but it is clearly def not that

[worthy cape]

It might help to understand the problem it was made to solve.

[modest path]

Is it more than just CDN update efficiency?

[worthy cape]

1. there's too many files in the GGPK, leading to excessive fragmentation and I/O overhead.
2. Steam isn't great at patching large files, requiring a full copy of the file to modify it.

[modest path]

Yep

[worthy cape]

Bundling files reduced the entry count by like 10x for GGPKs, and changed Steam from having one monolithic golden GGPK to 60-70k files on disk.

[modest path]

Yep, vfs to split the large, consolidate the small, and hopefully make read perf tolerable for both

[worthy cape]

A bit more upfront effort to obtain any particular inner file, but you have probably gotten several other ones you needed loaded at the same time for free.

[modest path]

@worthy cape What are your manifests keyed by? Like what is their ID? Game version is the binary, is a manifest pointers to the state of the VFS at a specific timestamp or something?

And correct me if I'm wrong, but other than backups/replayability in case of bugs, from an archival perspective, the only thing that matters in the game data is the contents of hte VFS that the game would have access to at a specific timestamp, right?

[worthy cape]

I base my dataset off historical Steam changesets.

Three depots where changes have an unique ID, ordered builds which reference those depots.

There's different types of archival.
One may be interested in how the packs evolved, what hidden data there may be in the "garbage" parts of the file, etc.

[modest path]

I guess there's also the possibility of the vfs for some period of time only being compatible with clients from some other period of time given they're free to break back/forwards compatibility, so the contents of the VFS aren't just time, they're also game version associated to some degree

[worthy cape]

Or one may care solely about the logical files.

[modest path]

Def get this re: the tooling/strategies they use to make the vfs, I was more talking about the data in the vfs

Could def see a situation where there's some duplicate or masked data that isn't in the VFS but is in the raw data that could be discovered by looking hard enough

[worthy cape]

There's two sources of such data. The chain of FREE chunks in the GGPK, and orphaned FILE chunks that are not linked from any PDIR chunk.

[modest path]

Yeah, sounds about right

[worthy cape]

Freed data contains whatever file contents used to reside in that spot, it's not scrubbed between updates.

This all mostly to try to make Steam patches as small as possible, as each Steam user has the exact same reference GGPK that they carefully modify server-side.

[modest path]

It does seem like an api that gives you access to the contents of the vfs as it is understood, as well as the binaries that go with it - all at a particular timestamp, is a useful tool for looking at the game over time, but a much lower api that makes you deal with the raw data is more useful for really deep exploration

the former is basically a poe wayback machine lol

[worthy cape]

My own manifests are of the cooked flavour.
Lists all "loose" and "bundled" files for each edition of each depot.

[modest path]

edition and depot are steam specific lingo I assume

[worthy cape]

Depot is, "edition" I just made up as "version" and "release" don't quite apply.

[modest path]

gotcha

I assume the standalone client fetches bundlesf rom patch.poecdn.com - I also assume that steam uses their own CDN for depot downloads and does not touch patch.poecdn.com at all?

[worthy cape]

Indeed.

Completely disjoint delivery systems of the same data.

[modest path]

Same VFS state, different delivery, possibly different bonus data revealed in each if you stare long enough

[worthy cape]

My stuff:

My tooling essentially grabs the index of releases, obtains the file manifests, obtains files by individual content hash.

Got some neat tools that do things like symlink virtual game trees to files in a common storage folder; so tools that are built to operate on Steam installations work without change.

[worthy cape]

Since bundles were introduced, Standalone and Steam are identical content-wise apart from some additional bookkeeping in Standalone GGPKs.

The curious junk data is solely a Steam legacy GGPK concept.

[modest path]

Gotcha

btrfs has some block-based deduplication stuff that could possibly work on just-decompressed-but-not-unpacked data files

e.g. https://github.com/Zygo/bees

Dunno if it's possible to just decompress and not unpack sanely for all the various formats though

[worthy cape]

I'm allergic to filesystem-based deduplication after reading about ZFS's 😄

[modest path]

Hahaha

[worthy cape]

I'm generally happy with the application-side dedup I have through content awareness.
Steam downloads have easily accessible SHA1 hashes of data files.
GGPK has SHA256 hashes.
My own processing leverages SHA256 significantly for file identity.

[modest path]

nice

[worthy cape]

As bundles are stable, the hash of the bundle serves as a proxy for the compressed data.
Even if you decompressed all bundles, any deduplication oriented around blocks would probably not have too much success with the concatenated files within.

Say that you've got a bundle of files [A|B|C] in one release and end up with bundle [A|A2|B|C] in the next, at best you're going to get dedup on parts of A if it covers more than one block.

[modest path]

yerp

[worthy cape]

(unless there's some clever block shifting or whatnot, haven't read up on the algorithms as again, it's a big hog)

[modest path]

Doing content-addressable on the individual files is def going to win

[worthy cape]

In general, building Inya was very educational but I won't be able to put it into production for any projects.

I built it so it could serve as a data source for projects like PoB and the wiki, being able to work consistently with a particular game release as data source so that exports aren't just from "whatever files contributors have locally" and to not have to populate CI runners with full 50 gigabyte game releases.

While it was fun to build, I'm not built to operate web services with actual users and guarantees; I've already revamped formats and endpoints significantly.

There's also other reasons 😄

[modest path]

Oh I get that - I also have commitment issues ;) (also variable energy levels, and ADHD hobby timeslicing/obsession)

Mostly not even working on poe stuff recently due to friend being traveling and me working on random unrelated hardware projects, though I'm trying to get back into it

[worthy cape]

I've moved a bit farther away from the server I'm renting for this stuff, did you know that round-trip times matter when writing networked code? 😄

I look forward to getting the GRPC snout into production, as that can pipeline way better than HTTP sessions.

[frank drift]

https://web.mit.edu/jemorris/humor/500-miles

[modest path]

Classic

@worthy cape note that grpc2 /is/ http under the hood, it's just http2 which has good pipelining. You may be able to get some of the benefits if you verify that your entire path supports http2 and then makes ure you're sending requests to a pool that can handle pipelining

grpc and its client code is obviously well optimized for rpc use, so I'm not trying to say don't move to it :)

[worthy cape]

I've looked at HTTP2/3 for my stuff and it doesn't CF well and I can't seem to get session reuse and pipelining to work at all.

[lone ingot]

http3 is evil

[worthy cape]

GRPC seems to be a reasonable solution for now.

[modest path]

CF?

[worthy cape]

CloudFlare.

I've got the "production" host behind CF, thinking that the caching would make it scale better.

Probably a bit too girthy data to do much at all anyway, so might as well have a more intelligent service.

[modest path]

cdn caching is also going to increase in effectiveness with number of identical cacheable requests - you might not be hitting the rates needed to individual uris to get them to hold on to things

[worthy cape]

All rather moot as I don't have users 😄

[modest path]

fo sho

[worthy cape]

The server has unlimited gigabit, sufficient for all tooldevs as long as someone doesn't try to ad-hoc mirror everything.

[modest path]

I am fortunate enough to have uncapped gig fiber to my house, so I guess you have enough bandwidth for me alone ;)

@worthy cape what's the split on hash-addressable content vs metadata/dynamic/other on the production stack?

Like in GB

[worthy cape]

Maybe 300 gigs of addressable data, a hundred gigs of manifests and indices? A bit hard to tell as they're mostly compressed with zstandard.

I still don't have a good way to store the manifests in a way that they're queryable across releases.

A fistful of manifest files per release, no "vertical" queries possible.

I store these verbatim in the DB, compressed:

I've tried having tables with all the manifest entries, but the query times for anything are abysmal and any ancillary indices take up way too much space.

[modest path]

Given they're hash-addressable, it's effectively a key value or object store - google cloud storage costs $5.90 a month to store 300GB. And as long as you never need to iterate or explore there's no downside to small objects in GCS

hmm

[worthy cape]

Would kind of want to be able to ask for "all .tdt entries for all releases" or "all different versions of data/stats.dat64".

[modest path]

gotcha, yeah

[worthy cape]

I tried storing the bulk data with minio for a S3-like interface. Didn't scale at all as it uses the filesystem 1:1 under the covers 😄

[modest path]

yuck

[worthy cape]

I like having a single server with no additional costs attached, just 50 EUR/mo flat.

[modest path]

what the shit is the point of an object store if you're backing it with a filesystem

the whole point of object stores are to remove functionality that filesystems have so you can be cheaper/more durable/simpler

[worthy cape]

If I had more hardware and need, I'd jam it on Ceph object store or something.

Despite Ceph being meh 😉

[modest path]

Pretty sure ceph isn't so meh that its object store is a filesystem lol

[worthy cape]

Anyway, the bulk data isn't too bad now, pipelining and local caching solves most problems. It is also web tool friendly as I have HTTPS endpoints for casual web app use.

[modest path]

nice

[worthy cape]

The manifests I'd like a bit less bulky, they tend to blow up clients good.

[modest path]

Super cool project - And thanks for walking me through the bundle shit; that will def help if I try to push on the CDN slurping with native rust

[worthy cape]

100 megs of JSONL tends to multiply a lot in a client, and if you need to touch multiple releases, you need many gigabytes of indices and a local DB to query it meaningfully.

Been thinking of having some endpoints with limited query functionality like outlined before, but that means that the DB needs the data in a more queriable form.

[modest path]

@worthy cape if the jsonl you've got is all one schema for every line in a file, you might consider storing it in a columnar format like avro or parquet - They compress tremendously better due to the columns being adjacent. Also much more queryable directly

I mean the cells in each column being adjacent

I'm sure there's non-cloud ways to query parquet (bigquery supports it natively out of cloud storage)

[worthy cape]

Possibly. I generally have no idea what I'm doing, I'm a low level C++ lad at heart.

All this web and persistence stuff is on the edge of comfort.

[modest path]

Sounds like me with c/c++

I mean, don't get me wrong; I love mouthbreathing while staring at ghidra. I'm just not any good at it

[worthy cape]

You made me look at my postgres database.
I actually store the file manifests in two forms - the aforementioned JSONL, and with blobs of sqlite3 databases.

[modest path]

yo dawg

[worthy cape]

The latter being an experiment to whether it makes sense for a client to download a release manifest as a DB it can just attach in memory and query; instead of eating a gigabyte of memory on dicts.

The database VM has 8 GiB of RAM, runs on spinning HDD disks and has indices way larger than RAM.
Why is it unperformant? 😛

[modest path]

It's a mystery

@worthy cape do you have an example you can upload of a moderately sized jsonl file (10s to 100s of MB maybe?) I can look at?

[worthy cape]

https://inya.zao.se/poe-data/idxz/238961/8425158302642600964/bundled

(idxz endpoint gets you zst-compressed, idx gets you uncompressed)

One million rows, it seems.

[modest path]

slurped

[modest path]

https://i.imgur.com/1owPwdb.gif

[clear gazelle]

Is there any api to poe.prices that anyone is aware of? Would like to price my stashes once and store the data rather than forgetting what I did and when and repricing things

[modest path]

@worthy cape Ahh yeah - not much compression benefit of columnar as the only repetition in fields is in the path, which due to it being lexically sorted by that, tend to be nearby so compressing the json is getting them

if you have other data that has many rows with the same values in some fields it could be interesting, but not really for this

[worthy cape]

So you're looking at like 500k-1M entries per release, time number of releases (800-900?), which you'd like to make some interesting selections/stats from.

Can't normalize the DB much either and put paths in a separate table or something, as the joins also become brutal.

Had some queries that took like half an hour 😄

[modest path]

bigquery is built for shit like that, but yeah, not on your own server

One interesting thing with BQ - You can allow other people to have read access to the data and the query costs go to their GCP account

Kinda neat from a providing access to others point of view

[modest path]

@worthy cape What are 238961 and 8425158302642600964 in that path btw?

[worthy cape]

The depot for game files and the gid that currently refers to the latest release.

I've got one loose and one bundled for each data depot, only loose for executable ones.

You can see there that the two latest builds have the same data files and differing windows/mac executables.

[modest path]

@worthy cape btw, duckdb is very sqlite-like but has really good json import support: ```[vitaminmoo@winders-obsoleet-org ptest]$ time duckdb bundled.duckdb -c "CREATE TABLE bundled AS SELECT * FROM 'bundled.json'"; time duckdb bundled.duckdb -c "SELECT path, size FROM bundled ORDER BY size DESC LIMIT 10"

real 0m0.927s
user 0m3.433s
sys 0m1.241s
┌──────────────────────────────────────────────────────────────────────────────────────────────┬──────────┐
│ path │ size │
│ varchar │ int64 │
├──────────────────────────────────────────────────────────────────────────────────────────────┼──────────┤
│ art/microtransactions/hideout/kalguuranpianokeys/textures/kalguuranrunes_glow_bc7premult.dds │ 44739492 │
│ art/models/mapdevices/eldritchmapdevice/fx/mb/root/ground.fmt │ 34616196 │
│ data/spanish/mods.datl64 │ 34009872 │
│ data/german/mods.datl64 │ 33583828 │
│ data/russian/mods.datl64 │ 33535840 │
│ data/portuguese/mods.datl64 │ 33465584 │
│ data/french/mods.datl64 │ 33278348 │
│ data/thai/mods.datl64 │ 33083300 │
│ data/mods.datl64 │ 33015208 │
│ data/japanese/mods.datl64 │ 32792376 │
├──────────────────────────────────────────────────────────────────────────────────────────────┴──────────┤
│ 10 rows 2 columns │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────┘

real 0m0.043s
user 0m0.118s
sys 0m0.058s```

[worthy cape]

Heard someone recommend it the other day, might've been you 🙂

[modest path]

Yeah I was trying to use it for something else, didn't end up working out, but I was impressed

the duckdb file there is 161MB, so not as small as zst, but smaller than the raw json

and obviously is queryable

[worthy cape]

Can't wait for PoE2 to ruin all my scale estimates.

[modest path]

Hahahah

[worthy cape]

At least one modern game's worth of bulk data that's probably not shared with PoE1 in any way; I've got some volumes to resize.

[modest path]

At least storage is cheaper than it used to be

I just had two 12TB drives start having badblocks and it makes me sad - They aren't trustworthy, but they still "work" just fine

[worthy cape]

The server I've got is a 3000-series Ryzen, 64G RAM and 2x2T spinning rust HDD.

[modest path]

So now they're just sitting near the box their 24T replacements are in

[worthy cape]

Considering that I've got quite a bit of working set locally of releases when reingesting and developing the algorithms, it's kind of cramped by now.

[modest path]

def sounds cramped for what you're working on

[worthy cape]

Good thing about HDDs is that you have to actually think about I/O patterns.
Bad thing about HDDs is <see above>.

[modest path]

ram is the new disk

[worthy cape]

At home I've got 18T and 20T mirrors that have pristine copies of most releases, with post-GGPK releases inside ZIP for better filesystem performance.

Looking to build a new home server with some 2-4T SSDs so I can do some more database-oriented development.

It took months to download all those relases when I started a number of years ago.

[modest path]

I bet

Reminds me of downloading linux isos on dialup in like 2001

Took a week

[worthy cape]

In any way, it's a good thing we have enthusiast users like you that can carry the torch onward.

PoE tools rest on surprisingly few backs.

[modest path]

lol

I'm still a spring chicken at this - Though I do think I played more vscode than poe this league

[worthy cape]

[modest path]

You got ~10ish more bundle urls, and maybe the corresponding human version strings for the uids in text format you can paste me?

or just like, two whole releases worth or something

curious about how much duckdb saves if i load data with overlap in

[worthy cape]

https://inya.zao.se/poe-meta/builds/public
https://inya.zao.se/poe-data/[idx|idxz]/{depot}/{gid}/[bundled|loose]
https://inya.zao.se/poe-data/[cad|cadz]/{addr}

Play around, not guaranteeing that it keeps working over time.

[modest path]

Fo sho, I'm just messing around atm, literally zero idea of useful things to do /with/ the data

[worthy cape]

Endpoints with z get you zstandard, idx/cad gets you release manifests and file contents.

[modest path]

You might consider setting a content-disposition on the endpoints so tools know what to name files when you download them - as is wget and friends will just create files named 'public', 'bundled', 'loose', etc

[worthy cape]

I do.

curl -OLJ will do-the-right-thing, as does browsers.

[modest path]

oh wget doesn't use content-disposition by default does it

is manifests.<depot>.download the {addr} field above or something else

[worthy cape]

addr being a content hash for any file you want to obtain.

{"path": "PathOfExileClient.app/Contents/PkgInfo", "sha256": "82502191c9484b04d685374f9879a0066069c49b8acae7a04b01d38d07e8eca0", "phash": "7624680036178654426", "size": 8}

https://inya.zao.se/poe-data/cad/82502191c9484b04d685374f9879a0066069c49b8acae7a04b01d38d07e8eca0

Gets you 82502191c9484b04d685374f9879a0066069c49b8acae7a04b01d38d07e8eca0.bin containing the contents of that PkgInfo file.

(poor example as the file has uninteresting contents, but you get the gist)

[modest path]

Oh! obviously

[worthy cape]

(cad as in content-addressable-data)

size and download in the builds endpoint are just the size that Steam claims the manifest occupies, on-disk and on-the-wire respectively.

[modest path]

gotcha

@worthy cape if the gid of a manifest is the same its' the same content? or is the depot part of the actual id

[worthy cape]

I've assumed that manifest GIDs are globally unique across depots.

You use them to identify what snapshot of contents you want in a particular depot. It's highly unlikely that you see one used in another depot as well.

https://steamdb.info/faq/#manifest

SteamDB has a nice glossary.

As you can see, a lot of this is built around a single game; would be annoying to jam in however PoE2 is going to look in this.

Good thing I probably don't have to bother.

[worthy cape]

Kind of alluring still to have a compact dataset in the closet or locally on a dev machine tho.

As a rule of thumb, DuckDB requires a minimum of 125 MB of memory per thread. For example, if you use 8 threads, you need at least 1 GB of memory. For ideal performance, aggregation-heavy workloads require approx. 5 GB memory per thread and join-heavy workloads require approximately 10 GB memory per thread.

[modest path]

Yeah I have no idea where the limits of this thing are

[rapid pagoda]

it was for a very long time actually, filestore was usually backed by xfs

[modest path]

That's like implementing a hammer by putting your cat in a burlap sack

[empty knot]

Hi guys, how are you doing

[modest path]

That's gotta be a bot

[flat star]

🆘 GET Profile Chars

Hello everyone!
I wanted to implement a "Whisp" button on the site (where the message is copied to the clipboard for pasting on the site), but I encountered the fact that the message needs to be addressed by character name (it is not possible by account name). To get the character name (current or any from the profile), you need to make a request via OAuth GET /character (account:characters) and take the name of any character or the active one (current = true) from the list.

But do I understand correctly that only the account itself has access to the list of characters? And how do you get the list of characters from another account? For example, like POB does

[flat star]

🆘 GET Profile Chars

[lone ingot]

any idea if there is any representation of item filter attributes in the game files?

the spec for filters is https://www.pathofexile.com/item-filter/about (I think). Is there any hints or deterministic properties that show the possible conditions for a base type?

e.g. a chaos orb does not have an item level and cannot be scourged or corrupted. any way to prove that with data?

asking now before I go ahead and do the entire mapping myself

[flat star]

any idea if there is any representation

[worthy cape]

We can see some tool excerpts in this new video - nothing super fancy: https://www.youtube.com/watch?v=uDBdEPohbh8

Mostly art-oriented things it seems.

[lone ingot]

progresssss https://i.gyazo.com/4629fdd2337ee2d4011bcd6518f75f81.gif

and https://i.gyazo.com/3f539577b8377a5b427894147b730b86.gif

pretty nice feedback loop for filter generation I think 😄

(dont look at the icons @worthy cape i still havent fixed them)

[worthy cape]

[flat star]

Tell me where I can find the coordinates of passive rocks, their names and hashes that are stored in the character's profile? And tell me where it is determined which point is associated with which (you need to draw a tree on your site)

[rapid pagoda]

The data for the passive and atlas trees is available at https://github.com/grindinggear/skilltree-export

[frank drift]

there are many open source implementations of the tree if you need a reference

[bold mulch]

Hello

Could anyone tell me how to join guild.

[worthy cape]

There is no tool or automation for it. A guild leader takes a character name and sends an invite in-game which the subject gets to accept, much like a party invite.
If you're (obviously) looking for a guild to join, use the social channels or forums to find people you want to be with. As guild slots have a low cap at like 250 and cost money to buy, many guilds are small and with good mates.

[bold mulch]

@zao got it, thanks'

[worthy cape]

I keep forgetting that you write through translators, "rocks" is a funny word to use for something like "keystones" 😄

[stoic locust]

i mean, keystone is a rock

[fickle yew]

@mortal bone Could https://github.com/EmmittJ/SkillTree_TypeScript get a LICENSE file? I would assume it is MIT like PoESkillTree? 🙂

[mortal bone]

Oh, my bad, yep 🙂

the package.json has the license, but I will put the file in there

[mortal bone]

done 🙂

[fickle yew]

Ah, didn't check package.json. Thanks!

[desert turtle]

Is https://github.com/meta-is-beta/horadric-helper still a viable way to display tooltips for items and such on websites?

[golden knot]

Last commit was 2 years ago and the blog website is gone. Seems abandoned unfortunately. it was a really cool effort

[desert turtle]

Yeah, that was my fear.

Are there any other projects out there like this? Or are sites that have such implementations using custom code?

[formal kraken]

I have something like it in pobbin, but nothing you can just use

[golden knot]

The wiki generates item hover boxes, but also probably not something you can just pull and use

[formal kraken]

https://github.com/Dav1dde/pasteofexile/blob/master/app/src/components/pob_item.rs almost usable I guess

[desert turtle]

I'm more interested in just pulling item information rather than real-time/inventory data, similar to how poedb and poewiki do it. From the looks of it, the pasteofexile code looks like it's pulling from items in-game?

[formal kraken]

Sounds like you have 2 questions, how to get the data and then how to display it.

What data are you interested in?

[desert turtle]

I believe getting the data is just using the fork of RePoE and PyPoE? Then, displaying the data on a semi-static site (I'm using Astro) to display it in tutorials and informatative topics.

[formal kraken]

Sure, RePoE works.

[desert turtle]

I'll be needing data for almost everything, uniques, gems, etc.

[formal kraken]

For displaying you'll probably have to do your own frontend thingy but it's not that bad. You can also still try to use horadric helper, nothing really changed.

[desert turtle]

I'm going to horadric helper first. I'm a tech writer first, aspiring developer second 😅

Thanks for the tips! I'm sure I'll be around asking questions from time to time!

[loud flint]

I implemented this, which I will open source

All in React tho

Based on data extracted from @klayver/poe-itemtext-parser right now, but will write my own parser soon when I have a bit more time

[desert turtle]

I can use React on my Astro site 🙂

Excited to see what you come up with!

[loud flint]

Will publish the package either later today or tomorrow

[loud flint]

https://github.com/Path-of-Tools/poe-item-hover-react published, not fully tested yet in different projects so things might still be a bit janky

Current caveats:

• Doesnt support crucible items (my website supports it, but the standalone item hover does not yet)
• Only supports rare items (adding support for magic and unique soon)
• Doesnt support CTRL - ALT - C (limitation of parser, will be fixed when I write my own)

Also oh my god, publishing a React library is 10x worse than it should be

[worthy cape]

[loud flint]

I really need to open source more of the things I made, but publishing it means I also have to support and update it

[worthy cape]

You break it, you get to keep all the pieces.

[frank drift]

nice petter

[lone ingot]

MIT license aka you use it, your problem

😄

[mint karma]

thats awesome, good job!

[golden bane]

David Beazley has this Q&A in most of his repos:

Q: Who maintains <project_name>?
A: If you're using it, you do. You maintain <project_name>.

[mint karma]

I've started working on magic item regex as a result of trying to solve it generic for tinctures!

[mint karma]

Hopefully I'll do some decent progress and have a MVP tomorrow of rolling some types of magic modifiers on items

[loud flint]

Lol, good one

[loud flint]

First production site using the library above

Not super happy with the spacing and stuff, but it gets the job done

[compact wigeon]

Good and existing is better than perfect and not existing

[lone ingot]

worked well, ty @worthy cape

aspect ratio is bricked on the screenshot

rogue windows update has completely bricked my pc, it's pretty awesome

I used https://www.npmjs.com/package/sharp for the img stuff in node, worked fine. that's how you do it if it's useful to anyone

async function fixIcon(filePath: string, height: number) {
  const img = sharp(filePath);
  const meta = await img.clone().metadata();
  if (meta.width !== 236) return;
  const one = await img
    .clone()
    .extract({ left: 0, top: 0, width: 78, height })
    .toBuffer();
  const two = await img
    .clone()
    .extract({ left: 78, top: 0, width: 78, height })
    .toBuffer();
  const three = img.clone().extract({ left: 156, top: 0, width: 78, height });
  const buffer = await three
    .composite([{ input: two }, { input: one }])
    .toBuffer();
  fs.writeFileSync(filePath, buffer);
}

[formal kraken]

One day I will implement alt gems correctly

[mint karma]

Nice @pseudo ocean , sounds like a cool project. So its more of a db lookup tool with support for regex?

neversink has some real good stuff in this public repos that might help you out

https://github.com/NeverSinkDev/Filter-ItemEconomyAspects
https://github.com/NeverSinkDev/FilterBlade-Public-Assets

[mint karma]

its not pretty, but we're getting there!

[loud flint]

Hot!

[mint karma]

now for some testing 😄

[loud flint]

Looking good! Happy to do some alteration spamming if you need QA

[mint karma]

I appriciate that!

I'll do some more testing myself before you have to waste your precious alts. I will keep working on it tomorrow, but after I'm done with that, getting some QA would be awesome 😄

The "beta" is out at https://poe.re/#/items currently, but expect a lot of bugs since I haven't really tested it

[loud flint]

Will take a look!

Also, innså akkurat at du er med i Norsk Programming, var Moderator der fra... 2019 til nå nylig da jeg valgte å trekke meg 😂

Sorry for non-English other peepz

[mint karma]

That's great, yep, currently living in Oslo. Never been very active at that discord, but it has some good vibes

[loud flint]

Lived in Oslo for quite a few years, but turned expat now, living live in the Netherlands lol

[worthy cape]

at least two good things to come out of Norway 😉

[worthy cape]

Nice to see Nordic peeps active in the community, back in the days it was mostly me and zensei from Sweden.

[loud flint]

The reason there's so many Scandinavian programmers is because it's so damn cold there's nothing better to do

But yeah, love to see it

[compact wigeon]

Us scandinavians keep moving out of scandinavia though

[mint karma]

Yo @compact wigeon , when are you moving out?

[compact wigeon]

Plane leaves on thursday the 24th

[crude bane]

I moved from Norway to Sweden, couldn't escape scandinavia 😦 Maybe next year tho

[loud flint]

Facts! Turns out it's way cheaper to live anywhere else in Europe, also closer to actual things going on

[worthy cape]

He doesn't know ^_^

[loud flint]

I don't know

[hazy fiber]

if only it was that close

[formal kraken]

Can confirm 2.5 weeks in norway were quite expensive

[loud flint]

I feel like there's something I don't know here

[worthy cape]

Oh, sorry to leave you hanging. Not even in Schengen now.

[loud flint]

I mean, there's plenty of places outside Schengen that are still more central than Scandinavia 😂

[worthy cape]

It's currently 01:26.

[loud flint]

New Zealand enjoyer?

[worthy cape]

[loud flint]

Zao confirmed inbound for GGG

[worthy cape]

Halp. I'm starting to kind of like Rust again. It's too soon for another cycle.

[worthy cape]

Been [t]here since August.

[loud flint]

Oh sick! Congratulations

[formal kraken]

🦀 give in 🦀

[loud flint]

I thought I wouldn't like Rust, then I used it for a very specific use-case at work, and it was the best experience of my life

I'm a crustacean now

[worthy cape]

[compact wigeon]

I'm the one that's inbound, starting the 28th, so yeah it's a bit farther than europe

[loud flint]

Congratulations to the both of you! What are you doing (Zao), and what will you be doing (Lily)?

[worthy cape]

UI/tools/gameplay programming for me.

[compact wigeon]

I'll be doing data analysis

[loud flint]

Cool!

[worthy cape]

My community tools like Bun will probably continue to work in the future, unless something changes.

[compact wigeon]

I expect to be releasing tree updates for path of pathing, but I have no idea if I'll do more work on the algorithm or not

Might not be one of the first sites to update with new trees anymore though, especially if there's any breaking changes to the tree structure

[mint karma]

How in the world do I explain to users that I can't match open prefixes on base types where the name is not two words long?

I can probably implement so they can choose the base they are rolling on, but I feel that breaks UI too much

[worthy cape]

Good old Vaal Temple Map 😄

[mint karma]

haha yep

[loud flint]

Crucible mods implemented Now time to do something about spacing and kerning

[mint karma]

I think I'll just leave that warning there, and say screw it until people start complaining too much

[mint karma]

huge!

Is the tool only for hover, or is it possible to "embed" the view too?

[loud flint]

You control yourself when it's visible, but it has some hardcoded values that makes it better for hover right now

[mint karma]

that makes sense, one usecase I could possibly see for poe.re is to have a few items (thats always displayed) to show examples of stuff that will match

[loud flint]

Not a bad suggestion at all, honestly. I was a bit tunnel visioned on my specific usecase but having fully embedded would be nice as well.

I'll take a look!

[mint karma]

yeah dont stress about it, but that could be useful

[mint karma]

A very reasonable time to be still up! The best time at the day is after 01:00

[loud flint]

100%, definitely a night owl myself

Going to bed between 4 and 6 AM every day

[mint karma]

Does these warnings make sense at all? I'm too deep in the code to know

This is because matching an open affix doesn't matter if you don't care about prefix/suffix.

better, I think

**Match item if either prefix or suffix is found
**
Warning! Using this setting together with 'open prefix/suffix' will match an item as long as it has an open prefix/suffix.

even better 😄

[loud flint]

Nice!

[lone ingot]

people that havent done stuff with text rendering don't understand how much of a pita it is haha, will gladly pinch your work once it's finished @loud flint

[loud flint]

Support for embedded (aka no hover) added, + magic/unique items 👀

Need to polish a bit (see extra separator on magic item for example) then I'll publish package @mint karma

Also, sorry for ping @compact isle, just want to make sure I'm not breaking any ToS here. Am I allowed to use these assets for my open source package? I know for sure the font since it's a generic font, but the separators and headers are technically yoinked from GGG

[worthy cape]

[loud flint]

I'm assuming it fine, but I'm also notoriously bad at understanding licenses

[frank drift]

they've never minded before

if you ask, you probably just get the usual copy/paste response

[loud flint]

I'd rather ask and get the usual copy paste response than not ask at all

[worthy cape]

I'd muse about it, but I'll leave it to the experts 😄

[loud flint]

Zao musings!

[compact isle]

the legal answer is that you're violating copyright by doing so

[loud flint]

[compact isle]

this has recently come up with some other stuff so I'll look into how we can provide those assets properly

[worthy cape]

i like turtles

[hazy fiber]

if your doing it as an open source package you could use a placeholder and people can just replace them with ggg assets if they want to run the risk themselves

[loud flint]

I like the way Riot does it with the Data Dragon, with clear stipulations on how to use the assets, what's okay and what's not, but completely understandable that this is a more complex matter

[worthy cape]

Historically, asset use has been ignored for well-meaning community tools projects, particularly sourced from the CDN.
If you're selling 3D printed models or t-shirts on Etsy or use them in a game or unrelated project, then right to jail dot gif.

[worthy cape]

I would honestly not be able to tell you the difference between the true art for tooltip borders and a reasonably similiar mockup 😄

Things like flask/gem art and unique items is harder to mock, but those are also things you could get off the CDN if you've seen the item on there before.

[loud flint]

Yeah so I think there's basically two options here

a) Use placeholder assets allowing people to inject other assets if they want to run the risk
b) Move forward assuming historic precedence on fair usage, and then take down the project upon request

Also, can't figure out how to send modmail for tool dev role :( Not sure I qualify only having a one hit wonder under my belt

[worthy cape]

The idea is that I as deputy tool developer would poke a relevant mod whenever I felt that someone deserved the roles; a role I haven't really kept up with.

Been meaning to suggest you.

[grave wren]

I'd really like to know about asset use as well, could easily remove the class pics from my discord bot embed if that's an issue or is that something i should contact support about

[mint karma]

Wow thats huge. Great job on this.

[crude bane]

Finally getting to a point with my market tracker website. It aims to fill in a gap that poe.ninja can't help with. There are some uniques that depend so much on their rolls and what modifiers are present for their value, such as Sublime Vision, Watcher's Eye or Forbidden Shako. The whole project started with me being frustrated with the highly unstable price of high demand Impossible escape jewels.

I'm also hoping, in the future, to track popular synth implicits and high value fractures. That depends on server load and motivation going forward though, but it will never try to directly compete with poe ninja. This project took way longer than I imagined (feels like we have been 90% finished for the last 5 months), but that's what happens when neither of us knew much about developing.

(The prices seen in this demo are not real)

[worthy cape]

I'm curious, do you surface any confidence in the listings there from volume, or is it all so low that it's buyer beware for all?

[loud flint]

I'm just assuming here, but would make sense for a lot (read: all) of these items are very low confidence on due to how rare they are - so I would lean towards the latter one

But there should probably be a visible disclaimer either way

[oak estuary]

Filtering mods to only those available on the selected unique item is a great QoL feature for less experienced players, nice!

[crude bane]

Confidence indicators are certainly something I want to show. But getting the basic functionality to work has been a priority. Currently we do have some protection against price fixing and outliers, but I havent been able to test it much

While I would mostly use the tool for low supply items, I also want it to be useful for league start planning, where you just want to know how much a certain 1 mod watcher's eye costs during week 1

[crude bane]

The same thing happens for base types 🙂

[gilded tree]

I only have a couple of them listed on .watch because for most it’s not exact as of the low volume like @worthy cape mentioned

But nevertheless, seems like a cool project. Gl!

[crude bane]

The website also comes with an API that allows you to do queries without going through the website.

Most of the codebase is actually public, so feel free to check out https://github.com/Path-of-Modifiers/pathofmodifiersapp. Would love to hear how things could be better, hopefully the structure makes it ok to find what interests you

[nova yoke]

Are there any guides on how to get started getting information out of the api? In the getting started section of the developer docs it states, "We are currently unable to process new applications." when referring to registering a new application. Are no new apps welcome or is there a different approach?

[hexed mesa]

application registration is handled manually, so my best bet would be that people responsible for it are busy with poe2 api and/or website stuff

[frank drift]

good bet

[nova yoke]

bummer

[worthy cape]

No alternative as far as I know, Piklo sounds about right.

[lone ingot]

drag and drop + sorting in arbitrary length trees? https://gyazo.com/98914490d58cb23021c5d00f6b95db80

implementation code for serialising to filter format was way easier than making that UI work, idk how you frontend guys do it

[formal kraken]

I hope you stick with this, I really want a filterblade alternative

[frank drift]

why

[formal kraken]

NeverSink is cooking too much, it's awesome, but also tlway too much. Also I have to tweak too much every new league which is a lot of work nowadays

[lone ingot]

if I am going to keep playing poe then I am going to finish this, planning to dogfood it for poe2/next league

[frank drift]

I barely have to tweak anything but I'm also a filterblade expert so

[formal kraken]

Most of the base configs are just not working for me. Bases, currency, recipes, leveling

[lone ingot]

it'll be open sourced also, I am just making a lot of changes at the moment and dont think it's worth having open yet. also need to work out all the licensing/copyright stuff haha

[formal kraken]

The presets regularly break

[lone ingot]

I think filterblade is great overall, espesh for newer and lower play time players. I just spend like 300-400 hours per league and have managed to brick my filter 3 times on filterblade lol (twice this league)

I quit the league and started rage programming basically, I was in ssf and just couldnt be bothered remaking such a fine tuned filter

[golden knot]

Are you wanting to write your own filters from scratch?

[formal kraken]

No

Well depends, if your tool makes it comfortable enough, I would.

[frank drift]

what did you do? overwrite a save?

[lone ingot]

my perspective is that the community just lacks a UI that's unopinionated and lined up with the spec of GGG's filters, that's coming from an SSF pov

I think filterblade is good for trade, I just dont need all the changes and potential to break

I can show you the bricked filter now if you want, Im prettly sure it's still broken in advanced settings lol

[frank drift]

I assume you posted something about it on the FB discord

[lone ingot]

I did yeah, many paragraphs

[formal kraken]

Also appreciate a local tool, that can update the filters directly. The bulk download is now kinda hidden on fb and the poe sync cant do bulk and is also too slow.

[frank drift]

ah you posted when NS and Zoey were on vacation, plus you didn't leave a FB link

drop a FB link on FB discord and I can get them to take a look if they have time

[clever thistle]

filter generation is soo sloow for me (FF)

[lone ingot]

that was also what I wanted @formal kraken , sometimes I screw up and need to send a second change through and the nature of ratelimiting means you cant do that

that's not anyone's fault, it's just the nature of public APIs

the last major thing was versioning, I like the idea of undo/redo and having saved states as an opt in (which is much easier and cheaper to do locally)

[frank drift]

FB supports local filesystem saving on Chrome, I do that when I want to make quick changes

[lone ingot]

can it save directly to an arbtirary path?

that's cool if it can, I didnt know that

Im using tauri v2 for chromatic (which I think will be its name, probably?) and Ive been loving it

[formal kraken]

Fb might be able to do it all, but I'm looking forward to your tool and a different spin on it

[lone ingot]

wish I was more talented at rust, I think there is a lot you can do in terms of perf having a non browser runtime. my plan anyway is to release something functional and see if anyone takes the bait to optimising 😛

the other thing that semi inspired me was the chaos recipe tool that dynamically updates your local filter, I thought that was kinda cool. in my mind there's a later feature of being able to set rules like "I want 5k transmutes and then I want to stop seeing them"

wealthyexile managed to work out an optimised way to retrieve/cache lots of a player stash and that is cloud based and his store, when you have local filesystem you can remove an entire factor of complexity in storage

[lone ingot]

I think so, I'd still recommend fb to most people, especially newer players

the opinionated take on tiering etc is really good for the 99%

[frank drift]

arbitrary path yeah

[lone ingot]

filter is still in the same state on fb now by the way, if you want to track it down and look at it I can help

forgot to tag you @frank drift

[frank drift]

like I said, leave a FB link to it on FB discord

[lone ingot]

oh that one, ta

[worthy cape]

I keep being continuously re-confused by the use of the term "FB", mentally expanding it to the book of faces.

[frank drift]

that's Meta now

[formal forum]

just out of curiosity does anyone know how poe.ninja/builds gets all the character data? there's no API that explicitly returns character data

[worthy cape]

There's ladders and there's explicit authorization via oauth for people who want their characters indexed.

[formal forum]

https://www.pathofexile.com/developer/docs/reference#leagues-ladder <--this one?

[worthy cape]

Probably, yeah.

[formal forum]

awesome. got something new to play with

[lone ingot]

making an undo/redo system, actually a really fun problem

[frank drift]

I've tried to convince Zoey to do one, she's not interested

[worthy cape]

Primary way I tested what happened when 32-bit PoB ran out of memory was to cause huge undo states to be generated.

[lone ingot]

I had no idea how complicated it is for nested complex objects lol, it's good though, making two stacks of deltas and just going to go back and forth based on input

[worthy cape]

I remember what I did now. You could generate hefty state by scrolling an item slot in Items in a build.

I just went ham and scrolled back-and-forth on like a jewel slot and it caused huge state ripples.

The only limit these days with 64-bit SimpleGraphic is your commit charge and imagination.

[lone ingot]

yeah we are so privileged now lol, Im using typescript/solidjs for the frontend and Im just kinda amazed at how fast it is for the stuff Im doing

[lone ingot]

false
false
true
true

[loud flint]

I was going to do that for my necropolis tool but never got around to fidgeting enough with it

[lone ingot]

I made something that's pretty generic in js/ts, I could share it

not rocket science either, it's just the command pattern + using a lib called microdiff. I am just storing the output of the lib in two arrays

having undo/redo between app launches was trivial also, just store the arrays along with the filter

[formal kraken]

now look into undo/redo trees

[lone ingot]

I was tempted but felt like I was jumping into one of those void holes in programming lol

I use it in nvim very occasionally, not for the faint of heart

think getting a ui (and keybinds) right for it is too difficult for average users so people just settle on destroying the redo stack when a user executes something

[severe flume]

@mint karma am i a big dumb dumb or is there no regex option for shields ?

nevermind i figured it out without the tool "scent | r Sto | g Rod | mmoth | ltering"

[mint karma]

Ah, I forgot to add that, I'll add it tomorrow 👍🏻

[compact wigeon]

Forgetting off-hand non-weapons is pretty classic (add quivers too )

[mint karma]

For sure, will add that too when I get around to it tomorrow, but currently I am very deep in factorio! 😄

[worthy cape]

I bought the game the other day to have something to pop in and out of to noodle with. First session ended up a slab of six hours <_<

[mint karma]

ahaha, its a very scary game, currently preparing to go to the second off-world planet

[severe flume]

anyone know what never sink's discord is ? the toolsite doesn't recognize me changing the minimum value or deactivating this rule as a change so it doesn't save with modified rule

@hushed relic

[frank drift]

it's linked at the top of FilterBlade

[frank drift]

(it was user error, for reference)

[severe flume]

was some random unnamed custom rule

no idea why that was there sincec there is already general single currency rule

but for some reason it was ( wasn't me i don't usually fk with the filter settings)

[mint karma]

This should be live now 🎉

[loud flint]

PEBCAC

[compact wigeon]

Looks fine!

[frank drift]

https://www.reddit.com/r/pathofexile/comments/1gdgr92/unplayable/

[worthy cape]

I love lopsided UI.

[hazy fiber]

I mean you see this with alot of things, mathematically centred/symmetrical doesnt always look the best to humans

[severe flume]

sweet, thanks !

[daring moss]

I wonder if this might also just be a technical limitation with how they're doing those groups + radii

[worthy cape]

Loose nodes with line connectors shouldn't be subject to orbital constraints, I reckon.

[daring moss]

Right, yeah, I agree with you on that in regards to the picture in the direct OP for that thread. I wrote my comment when I saw their harvest screenshot

[lone ingot]

that was a royal pita to get working, nested trees and managing state with undos/redos is hard https://gyazo.com/ebe0690cd40ee57392db8527eab69b4c

feels so nice now though

[unreal raft]

Did you manually "found" the groups? Or did you use the gamefiles? If yes, do you have that script somewhere?

[lone ingot]

game files > sqlite > query

I'll make this repo public eventually, you can use https://snosme.github.io/poe-dat-viewer/ to view the data and use the lib to pull the data yourself https://github.com/SnosMe/poe-dat-viewer/tree/master/lib

nothing that spectacular, I am just importing the json extracted with that lib, picking out the fields I am interested in and doing a bit of semantic mapping in the SQL query

it's not a very pretty query honestly, thankfully the dataset is very small so it doesnt matter 😛

also the format it comes out in isnt the greatest, I do additional stuff after to roll it up into the tree like structure

[tranquil token]

@velvet fog Could you update the KR, RU, PO, TH, FR, DE, ES database of POEDB the same as TW, CN, US, JP?

https://poedb.tw/us/Claws#VaalOrbCorruptedImplicit

For example, "% to Global Critical Strike Multiplier" on the link will appear normally in US, TW, CN, JP, but will appear as "Grants Level # Grace Skill" in other languages. In addition to this link, there are many other data that are different from the latest data.

[velvet fog]

updated

[chrome topaz]

hello, I published the old pob.party code. I see somebody made a new web one, but anyway maybe something on there would be useful https://github.com/xyzz/pob-party-oss

[carmine merlin]

@smoky nimbus ^might be on interest to you

[molten kernel]

does anyone know if the poeladder dev is here?

[worthy cape]

I've seen a few messages in the past. I see there's a contact email on the website.

[cobalt pollen]

making an atlas profit//strategy analyzer tool for fun... sweet lord I forgot how many basic currencies exist lol

[jaunty pulsar]

Hey there! I hope this is the right channel to ask. Since there's a version web version of POB does that mean we can change it's UI more freely? Possibly with HTML/CSS? Asking 'cause I'm a designer and I'd love to propose a UI revamp of some sort. Is this even remotely possible?

[hexed mesa]

as far as i know the web version of pob is more like "we run the normal pob in the browser" rather than actual web version. also given that its an archived repo its not exactly actively maintained thing.

(unless you are asking about something unrelated to the link some messages above, ignore me if thats the case)

[jaunty pulsar]

Hmm got it. Would you happen to know how feasible it would be to actually revamp the POB user interface? Would love to partner with a dev on this endevor but not sure how to find people who would be willing to do it

[hexed mesa]

https://github.com/PathOfBuildingCommunity/PathOfBuilding/blob/master/CONTRIBUTING.md

looks like theres semi private discord if you dm relevant people

[grave wren]

normal pob ui components are done as simple ui elements that can then be used in the lua code - its pretty far from the flexibility html/css would offer

you can find the UI elements here: https://github.com/PathOfBuildingCommunity/PathOfBuilding/tree/master/src/Classes

[compact wigeon]

There's been a few attempts at redesign, but they've never really gotten off the ground due to the way PoB is written.

[jaunty pulsar]

Yeah that makes sense. And it's fair people wouldn't want to rewrite the whole thing just because of the UI

Any chance to somehow connect your POB data to a web-based app made in React and then handle the UI there? Just use POB as a database or back-end of some kind? Is this too stupid? 🤔

[compact wigeon]

In theory anything is possible, but I would hesitate. It would be a large and involved process, in many ways it might be easier to do a rewrite, and we've had a few of those be attempted as well with varying levels of success. Its a very large project, you shouldn't underestimate the amount of work.

[languid flower]

hello, i'm trying to retrieve characters data from poe.ninja. But i'm struggling finding the API endpoints. I have been advised to use my web dev tool to see the requests made but without much success, I don't understand it :(
Anyone can provide me some informations about this?

[cobalt pollen]

this is such a fun project lol

Godot Engine is so well made for making poe tools x)

just using an entire top end game engine to make tools for a game I enjoy. Lmao

[languid flower]

@fickle yew sorry for bothering you :')

[lone ingot]

https://poe.ninja/api/data/currencyoverview?league=Settlers&type=Currency

use this as a reference, you need to set the league and other params obviously https://github.com/ayberkgezer/poe.ninja-API-Document?tab=readme-ov-file

[languid flower]

ah thanks but i'm not looking to gather data on currencies and items :s

I want to gather data on characters equipments, jewels, passive tree and the characters stat :0

this basicaly

and as much character as i can

[lone ingot]

the network request is called getcharacter when you load the page

[languid flower]

i have nothing D:

oh for a char

[lone ingot]

you must be filtering things out somewhere

[languid flower]

okay

[lone ingot]

yeah, load the character

[lone ingot]

honestly just filter on "api" and you will see them

the url is always poe.ninja/api/...

[languid flower]

okay beautiful i can get data for one character :D

[lone ingot]

try the same on the main page with above advice

[languid flower]

i get these

that's what i showed in my first message, i dont understand them, they are not json :(

[lone ingot]

oh that's clever

it's using protobufs

[languid flower]

i don't know this type of file, what are they representing?

[lone ingot]

it's just another way of representing the bytes being sent, it's much more efficient but more complex

[languid flower]

json are quite easy indeed x)

so should i maybe try to convert them to something i understand and maybe one of them hold the informations i'm looking for?

[lone ingot]

https://protobuf.dev/

there you go, that'll give you a good starting point :p https://github.com/lvlvllvlvllvlvl/poetrage/blob/main/src/apis/getMeta.ts

[languid flower]

ah there we go

[languid flower]

tyvm :D
i got some time to work on my project. I'll try to figure this out myself, i dont even know if thet data are in there but that's interesting :)

do you mind if i @ you once i made some progress?

[languid flower]

didn't know this tool btw :0

[lone ingot]

it will be, it's being received by your browser the same as json. json is bytes as well, your browser just knows to interpret the bytes and present them bc of the header that says Content-Type: application/json

you can see the header on the request you were confused about

[languid flower]

yea i see it

new file type unlocked i guess x)

and i like efficient stuff, maybe i'll use it next time instead of json objects :0

[lone ingot]

json is nice for easy integration between systems but it is a big file size, even when whitespace removed

for poe.ninja it makes sense to use an efficient strategy for their main search page

[languid flower]

100k+ characters rn :/

[lone ingot]

many people will be refreshing, sorting in all kinds of ways, etc. it seems like rasmuskl has set it up behind cdns

i'd be curious to know how it works honestly, it's pretty fast for what it's doing really

[languid flower]

for sure! id love to understand aswell

it's so exiting i can't wait to gather and work with the data :>

[hazy fiber]

I have done this for a few protoypes as well, and have been using a custom PoB thats converted over

but yeah its way overkill for most things

[hazy fiber]

the UI really isnt that bad to work with, the thing it needs the most is a proper container system, which I have prototyped and might finish up if an actual need for a UI re-write comes up, but at the same time theres people redoing PoB fully in other languages

Lua is used alot for UI related tasks in other applications, so it should be fine for PoB, what kind of things exactly are you after redoing (in lua or in some other UI framework like React)?

proper container system
as in elements which control the location of other elements, rather than needing to specify pixel locations for them, this is done by the auto elements like config tab and calcs tab, but shouldnt be too hard to implement elsewhere

[lone ingot]

lua for UI just sounds painful, not gonna lie

it's a great scripting language but I cant imagine what it's like compared to html

[brittle oasis]

You think we're going to see multiple PoB2's coming out? 😱 or more likely an updated PoB for PoE2?

[worthy cape]

I wish I had time to advance my runtime work for PoB, the already glacial pace on the font replacement project has gotten even slower now. I'll most certainly be on the sidelines cheering peeps on for EA and 3.26.

[hazy fiber]

I give it a 20% chance they actually finish it

[hazy fiber]

but we will see what happens for poe2, I assume somone will just take current PoB and hamfist poe2 data into it

[worthy cape]

There's definitely a place for a reworked runtime and application, but it's huge amounts of work to do so it still needs to retain some of the aspects that have made it so flexible and rapid to iterate on over the years.

You want something that still supports the ability to reshape it when entirely unexpected things change every 3-4 months, with results within hours or days.

Development is very bursty and often aligns with the hype cycle.
While you have contributions all throughout, it's always a bit slumbering inbetween the bookends.

Not always super conducive to fundamental improvement unless someone gets Great Inspiration.

[compact wigeon]

Plus it's not conducive to larger changes in general. Anything that takes longer than a single league to merge becomes outdated and easily lost

[languid flower]

hello, so i have found a tool online to decode the protofiles and in one of them i can find the names of some characters visible in the front build page (and some other i can't seem to find).

So my guess is that this file retrieve the name of the characters which fit the filters and the name is used to do the query for a specific char after that

look if i filter with this, and that i retrieve the file which start with 7a5ca9 ... there is only this character :)

so now i can may e iterate through them ?

[cobalt pollen]

SSF time 😄

[loud flint]

Anyone cooking something spicy for PoE 2?

[lone ingot]

if I can get access to the data I will probably be testing my filter tool thingy while I play

[frank drift]

NeverSink was requested to wait a month or two before releasing anything but the systems are theoretically ready

[worthy cape]

Nothing on the personal side, but I've been thinking about maybe hacking in some unicode glyphs on top of the sprite font in PoB, just to move that forward a bit.
Also took a copy of Inya's database onto a VM with SSDs and a lot of RAM, looking to see if I can make meaningful file index queries across PoE1 builds.
Inya doesn't support PoE2 much at all these days API-wise and I'd have to expand the disk space in prod to fit it.

At work, all the things

[velvet fog]

poe2 torrent when?

[compact wigeon]

I'm still waffling on how much I'll support PoP going forward. It depends on how the tree data files are structured for PoE2 and whether there are wild new features in them compared to PoE1, and if new features appear in either game's tree. Not sure how much time and energy I'll have to update things beyond just plugging in new trees to the existing code

[worthy cape]

20380119T031408Z

[velvet fog]

PoP typo?