tooldev-general

[worthy cape]

https://www.pathofpathing.com/?v=3.25.0-atlas

[velvet fog]

oh

[worthy cape]

Intelligent tree autocomplete from preferred/forbidden nodes.

[compact wigeon]

Tbf I have only myself to blame for people thinking it's a typo

But it's such a good name

[worthy cape]

PoPa?

[compact wigeon]

That'll just make people add another o in the middle

Path of Pathing is a great name, might just want to stop using the acronym for clarity's sake.

[oak estuary]

Love that people think it's a typo even when B or E is nowhere near P. "That rhymes with another common acronym, must be a typo"

[worthy cape]

"worm regards"

[velvet fog]

sorry about that

[compact wigeon]

Nah dw, you're far from alone in that

[clever thistle]

whatever floats yer goat

it's a doggy dog world after all, and everyone needs an escape goat for all intensive purposes, that's a gibbon

sorry, i'll nip it in the butt

[clever thistle]

(for peeps who aren't familiar: Whatever floats yer boat. It's a dog-eat dog world after all, and everyone needs a scape goat for all intents and purposes, that's a given. Sorry, i'll nip it in the bud.)

[grave wren]

it'll forever be goat now and you can't fix it

[mint maple]

How does the searches converted to URLs on https://www.pathofexile.com/trade/search/Settlers work? I wanna automate searching for all Forbidden Flame outcomes without price checking every outcome by hand (again).

I realise I could do it once and then save the URLs in a data file, but I'd rather avoid doing that as well

Maybe it's not public info, I just hoped it was

[grave wren]

press f12 when searching filter by xhr check how its done

[mint maple]

Ah, I'll start there

[frank drift]

it is indeed not public/not officially documented

there was at one point a page on poedb that had some info I think

[mint maple]

Ah. I should read that now that I know something I wanna do with it

[worthy cape]

You can construct a query as JSON, pass it to the trade website and get one of those trade URLs in return, or an error telling you that your query is too complex.
A lot of things around the trade site is monkey-see-monkey-do, generally not intended for user-less automation. The more kosher consumers are tools like APT and PoB.

[brittle oasis]

PoB uses the trade site? O.o

[frank drift]

it has a trade search function yes

[worthy cape]

There's a "Trade For These Items" kind of feature, and also integration to search for Timeless Jewels.

[brittle oasis]

Pretty bad ass, you give it a search url and it returns the best upgrade basically?

[frank drift]

you tell it what slot you want to look at and it returns a list of items based on weights you set

the list being a trade search

[lone ingot]

awakened poe trade is a good reference for it https://github.com/SnosMe/awakened-poe-trade/blob/master/renderer/src/web/price-check/trade/pathofexile-trade.ts

[velvet fog]

https://poedb.tw/us/API%3ATrade

[mint maple]

ty!

[mint maple]

I'll look at this, need to get better at I think javascript to understand most of it? but may as well learn more of that anyway

[mint maple]

is that what they show on the poedb page that chuanhsing linked? sounds useful

[worthy cape]

Yep.

The syntax is somewhat straightforward, you can get inspiration by making queries interactively on the website yourself.

[mint maple]

I'll look around. I should try to get something done with this now while I wait for the uni to handle paperwork on officialising my degree, state of poe1 is a bit of a shame but I have time now and need something to do

[pallid wraith]

Those links can expire.
Though it might be only for unused ones. I forgot if it's a set duration, or a duration after last use.

[frank drift]

~90 days without being accessed generally

but if the backend changes they can also all just get invalidated/deleted

two ways

basically do not count on any trade query working forever

[lone ingot]

just found the /itemfilter x command, it's like christmas

when did this get added

[worthy cape]

There's also /reloaditemfilter if you just wish to iterate.

[lone ingot]

it gets better

I had no idea these existed and Ive played since legion lol

[worthy cape]

/itemfilter appears in /help, heh.

I just found out that my Inya data pipeline has been broken since late August <_<

Monitoring is good, mkay.

[worthy cape]

The reload is mentioned in this post: https://www.reddit.com/r/pathofexile/comments/1aps72h/comment/kq857rf/; at least nine months ago.

[worthy cape]

If anyone relies on dat-scope for detecting any eventual changes for the new event, it's currently serving stale information.

Might get around reconstructing the missed changes from steamdb in the weekend.

Turns out that all my services were up and running, except that SteamWebPipes had lost connection to Steam and didn't restart, reconnect or terminate any listening clients; they just got nothing.

Not sure how to monitor that except maybe have a canary watchdog client that honks if it has no updates in a particular amount of time?

[grave wren]

sounds like the easiest way to just store the last interaction date and have that monitored

[oak estuary]

Did SteamWebPipes get a disconnected event or was that the part that silently failed?

[faint remnant]

Has there been any datamining on potential changes to the lantern?

[ashen aspen]

Hello, would it be feasible to build something like this for poe: https://github.com/SteamDatabase/GameTracking-CS2 ?
Just explaining what it does if it isn't clear: When a new patch is released, it downloads the game update and runs tooling on the files (like unpacking, extracting strings from binaries, translating game specific data into a redable text-format) and then commits the tooling output. each commit is a nicely readable diff displaying what changed from version to version.

I know there's a lot of tools (i.e. on github) that maybe do what I am talking about (krangler? changeapi?), but it is not clear to me what those tools do from reading the readme.

Or does the data format change too much between releases and you guys esentially end up manually hunting for offsets in dat files?

[hexed mesa]

lvlvl does that with repoe to some extent, as far as i can see https://github.com/repoe-fork/repoe-fork.github.io

[ashen aspen]

Oh, that's cool. Thanks, i'll have a look.

[oak estuary]

It's a lot of the latter though. We can automate how much each file changed and such, but specific data diffs require offset hunting. Even lvlvl manually updates columns and exports the result

[ashen aspen]

Thanks for clarifying.

[worthy cape]

Can't really tell, it makes some amount of noise to the console but I'm running it unmodified and treat it as a black box I can connect to with a client. It looks like it's an internal problem.
Really hard to troubleshoot as it's something that happens like twice a year, tops.

Might just be reasonable to restart the service monthly or something.

My tooling gathers and archives all versions of the game, theoretically making it all available to my tools.
The one tool I got around running persistently against that was one that looks at the very outermost layer of DAT file, telling if the overall file shape seems to have changed to make people aware that there's mining work to be done.

[worthy cape]

My tooling is somewhat of a reinvention of that, a bit more hardcoded to PoE and the fact that I grab all the files, including historically.
Monitor PICS for changes, record the depot manifests to get, pass it on via a web service to another tool that the downloader service pulls work from and obtains the data files.
Those are deduplicated and recorded in a data & metadata database, for future use.

Builds on some stock pieces like DepotDownloader and SteamWebPipes, but a lot of custom glue too.

Amusingly enough, it would probably be more robust if I just polled Steam every now and then; thought that SteamWebPipes would be able to run for extended periods of time.

Host uptime is just something like 200 days.

[loud flint]

I should really set up some tooling of my own for this stuff

I only have a ton of them for League of Legends, but none yet for PoE

[worthy cape]

I finally have some closet compute up and running now at home, so I could finally start iterating on this infra again.

Prod/dev is almost 300 ms away now, so terminal work suuucks 😄

[loud flint]

Lol yeah that makes sense 😬

Also no better excuse to have a homelab

[worthy cape]

Also probably better for the community to pick up the torch here.

[zenith zodiac]

for a slightly more raw view of the dat files there's https://github.com/lvlvllvlvllvlvl/dat-schema-validator/tree/develop/history, which uses a heuristic to guess what the unknown column types are. and l recently created this repo https://github.com/lvlvllvlvllvlvl/poecdn-bundle-index/tree/dev/output/patch.poecdn.com/, which just lists every file contained in the bundle - over 50mb of filename data, as there are more than a million files (each file path is relative to the one above, as explained in the readme, because when l output absolute paths the file size was over gh's limit of 100mb for a single file)

[worthy cape]

It takes my new path database around 45 seconds to do a full table scan on the table of every file entry.

(on a mirror of consumer SATA SSDs)

[zenith zodiac]

that's every file ever distributed right? is it a lot more than what's in the current bundle or do the contents stay mostly the same?

[worthy cape]

2.16 million distinct full paths stored, 1.36 million if you are case-insensitive.

333 million file entries total.

A few releases ago, the main depot had ~42k loose files and 1.07 million bundled files.

If I had to fathom a guess, the path churn is probably mostly in shader cache entries and bundle names.

Distinct data file contents: 2.73 million objects

Bundles and shaders probably skew this over time.

It would be fun to get it back up and running again, but I have limited use for it now.

[ashen aspen]

The tooling you're talking about here, is it publicly available? I see some stuff might be at https://github.com/orgs/pale-court.

[worthy cape]

Yeah, that's where I kept most of that project.

(Inya, the namesake of my system, is one of the four members of the Pale Council at their Court 😉 )

[ashen aspen]

I saw inya mentioned on your homepage. The repo here is pretty small tho and no readme :)

[ashen aspen]

I never heard of SteamWebPipes, but I wrote something similar that is just polling steam for updates and it's been running without issue for 6 months now. So from experience, this is a viable thing to do.

[worthy cape]

I was running into some rate limiting problems with just DepotDownloader and didn't want to make it worse, I think.

(completely un-Bing:able too, as it's drowned out by everyone and their dog trying to abuse Family Sharing)

[ashen aspen]

As in slower download speeds or straight up no responses?

Idk DepotDownloader either, but using https://github.com/ValvePython/steam, you can selectively only download chunks that changed between releases. Tho this might be of limited use for poe, depending on how the ggpk is structured.

[worthy cape]

The Steam release of PoE stopped using GGPKs around the Heist era when bundles were introduced.

[ashen aspen]

Oh.

[worthy cape]

There's a distinct split there, between distributing a golden GGPK that takes 30 gigs of disk to patch and shipping like 20-30 thousand bundle files.

DepotDownloader keeps track of what manifest it has in the staging area you point it to and applies delta updates.

[ashen aspen]

That's nice. Bet they did this exactly for this reason, to allow steam to only download what actually changed, keeping download sizes low.

[worthy cape]

The way I use it these days is to parse the protobuf manifest it keeps and use the SHA1:s in that to know what files I need to ingest.

[ashen aspen]

Ah, ok. So DepotDownloader does delta downloads too.

Yea got it. I do the same with steam python lib.

[worthy cape]

I could probably do something more clever if I could apply Steam's deltas myself, but I like my sanity.

[ashen aspen]

In your tooling, do you somewhere focus on diffing data from one patch to the next?
I think what ivy is doing with the repos they posted is pretty cool, being able to get an overview of what changed between releases.

I've been thinking about doing something like this for poe myself as I did this before for other games but am not familiar with poe regarding its data structure.

Assessing what's out there tho is kind of challenging given that there's like 100 repos on github that do something adjacent to this and most are badly documented or not documented at all (understandably so, given that those are personal projects).

[worthy cape]

Inya (with the krangler tool) focuses on obtaining and making available any fileset from any time in public PoE1 history, without regard to what the files actually mean.

It's supposed to be a good source of data to build other more investigative tools on.

I keep a dat-meta repository of JSON information on the overall shape of the game's DAT64 files, generated via dat-shape and visualised via dat-scope.

Data-wise, there's a ton of data file types in a PoE1 release.

There's a number of textual formats that define things like item base types and object types, in .it[c] and .ot[c] files. There's actors in .ao[c] files, AI scripts in .ais, etc.

The big bugbear in PoE land is the .dat64 files, with almost a thousand different binary tabular data files that are directly used by the game client to drive all sorts of mechanisms.

Their structure is that they first have a fixed section of fixed-width rows of data, which are implicitly divided into columns with a small set of data types.
These may either contain fixed-width data in-line or point into the trailing part of the file for variable-sized data like lists and strings.
Some of those fields are row references to other rows in the same table or to rows in other tables.

One of the big challenges to data mining is that the schema for those rows is unknown, neither the extents, name or data type of columns are known; a miner has to slice up a row of data into what they believe are fields based on how the data looks when they squint; and then try to figure out what meaning they have in the context of the file's name.

The destination of foreign references to other tables is also unknown, a miner has to guess based on the perceived purpose of the file and what other files may have as many rows as the values seem to suggest.

Large number in the many thousands? Probably a Mods, Stats or BaseItemTypes; etc.

There's a shared dat-schema repository that attempts to gather contributions to how the schema currently looks in order to share the burden between different toolmakers.

On the other file formats for the game like materials, meshes, and all sorts of terrain metadata - those are all in bespoke binary file formats that can take years to reverse-engineer - ask me how I know 😉

I can note that I built this in part so that I could do large scale file format comparisons to find and leverage any "rosetta stone" files that has the same data but with different versions across generations, to suss out what a file format version introduces or removes.

There's some binary file formats like terrain multitiles (.tdt) that are up to like version 17 and room files (.arm) are probably in the 30s or 40s by now.

Pretty much any game asset file type is has a version and older untouched files in a release are often in older versions of the format as they typically have no reason to be revised.

Sometimes there's releases where all the files are ported to a newer version, typically when there's some fundamental transition in the engine, like when all the mesh file formats were unified a bit in how they internally stored submeshes.

Terrain, static and skinned meshes used to be significantly different; now they're at least uniform for the geometry payloads.

These days I'm tainted when it comes to data mining, so the most I can do is play around for my own education and cheer all you peeps on.

[pallid wraith]

I have been wondering why you didn't get your role changed yet lol

[worthy cape]

I like being able to talk carefully without people screenshotting and misinterpreting every word. 😉

[pallid wraith]

I think the only funny role message I've screenshotted was praise, but yeah, I can understand not wanting to be treated differently after all this time

[ashen aspen]

Oh.. I understand, nice.

[ashen aspen]

Are changes to the format usually only happening with league releases or does this also happen with patch releases?

[worthy cape]

You've got three times when it tends to happen. League start, league minor patches, and in the lead-up to the next league.

It can happen in any client patch that includes new executables, as the code and data needs to be in sync.

That's why I built the dat-scope tooling, to be able to see at a glance whenever you feel like it whether something has happened that you might want to react to.

New rows in Mods/Stats/GrantedEffects/BaseItemTypes/ItemVisualIdentity/HideoutDoodads? Better take an actual human look.

It's way too common in community projects that effort and attention comes in waves before/after challenge league starts.

Things that happen in a .1 patch can be completely overseen, and even the stuff that happens in a MTX/pre-league patch can be surprising.

It helps if you have a solid grasp on the previous league's files before you start working on the next league's files hours before the league starts.

One could even make a RSS Atom feed of that kind of dat-meta information, but I never got around doing so.

[ashen aspen]

Okay I see. So the limiting factor here is time and people..

Or I guess idealy, ggg would release the schema.

[worthy cape]

Projects tend to focus on the types of data files that interest them, and there's quite a small set of people in the community.

PoB is going to be quite into skills, stats, items, and all that jazz.
The wiki and poedb is going to be way broader but maybe shallower.

Hideout enthusiasts like myself would care more about doodads and MTX.

[worthy cape]

In some way, the friction in data mining is a good thing for pace and not instantly solving/spoiling a league, but it is a bit cruel.

It's been years since I played in a league start, always on virtual call in the community for my tools and expertise across projects.

[pallid wraith]

And now it'll be on call, but on the other side?

[ashen aspen]

As long as that's fun and not a burden..

[worthy cape]

I don't see myself doing much in the way of firefighting.

[worthy cape]

I love a support role, enabling peeps to do cool stuff.

Healer at heart 😛

[pallid wraith]

Makes sense

[ashen aspen]

I am new to this side of poe, but it sounds like you have had an important role in regards to maintainability of community tools. Are you going to be able to continue doing this?

[worthy cape]

I didn't produce too much around mining tools, the biggest one is probably bun_extract_file bundle extractor and porting the ooz decompressor to more architectures and Python.

I unscrewed a whole bunch around PyPoE's data access for wiki mining purposes, and did a major upgrade of the Path of Building graphics engine.

Outside of that, I mostly served as a communicator and information spreader, connecting different projects.

I'm sure people and the projects will be fine and find their way, particularly as they may group up to tackle PoE2 together.

I want to continue improving PoB's engine whenever that mythical "free time" thing manifests, particularly as that's as conflict-free as it gets; just fun old-school engine work.

As for all the other file formats I've dug into, there's essentially no-one that cares about them and publishing research on them only really helps cheaters.

The most interest people seemed to have has been around model conversion for external purposes like 3D printing and asset use outside of PoE, neither of which I'm super keen to further.

Online model viewers are largely unfeasible due to the material system being extremely special, so that's a dead-end route too.

In short, old tools will likely continue to work as long as the data stays similar, and I'm sure I or someone can unscrew them if they break.

Probably not going to see me shouting from the rooftops that there's new and fun DAT files in release X anymore 😛

[ashen aspen]

I know this is getting more into personal stuff.. but as someone who has been working on reversing every aspect of another game for years at this point, I always wonder how it feels to suddenly have a way of just looking stuff up in docs or source code.
I always asked myself if I would still care. Like, the mystery might be kinda gone at that point.

[worthy cape]

I love being able to see people speculate and be able to dig into it and figure out myself how things work. It's a continuation of the journey and it's quite delightful to complete the previously unsolvable puzzles.

What's really hard is going from a decade of dissemination and collaboration to not being able to talk and share.

I get to channel all that gremlin energy into making things better for the studio instead, which I'm quite happy about.

For a lot of my format research, I had reached points where I had exhausted everything you could divine from data files, just leaving a ton of loose ends, where the tools I wanted to make just couldn't be built - like the Hideout Editor.

That thing with visual previews I wanted it to have required too much information that just realistically couldn't be obtained by reverse engineering.

[ashen aspen]

Time to learn assembler.

But there's not enough time in a human life ..

To reverse the whole binary ;)

[worthy cape]

It took me something like three months at the start of the year to get to the point where I could figure out the compression scheme used in one of the file formats; learning Ghidra and x86_64 disassembly along the way.

The program is just too big, which is a blessing and a curse.

I tried taking a look at a function related to drawing the 2D UI. It took well over a minute to decompile every time you named a variable or changed a data type.

At some point in time, it's just diminishing results and you're straying even further from the tools you were actually going to make.

Collectively got me a job tho 😛

[ashen aspen]

The "magnum opus" I was working on for my game was trying to reverse the net code. It is possible, but it's kind a a pointless exercise in some way.. and extremely tedious and time consuming.

[worthy cape]

I've never really had a desire for netcode or security, mostly because there's nothing meaningful you can do with it and any information you glean and publish is very likely to be used for ill deeds and cheats.

[ashen aspen]

Yea, I'm working on an FPS game. The goal there would be to be able to observe matches in a custom client, like a browser. In overhead view for example.

[worthy cape]

In the end, we do what we do to further the game and community - every game and environment is different.

One thing I like about PoE is that the community at a whole are good at adhering to the ToS, both in letter and in spirit, especially among toolmakers.

My aspirational dream tool was a Hideout Editor - one where you could edit hideout presets in a way more powerful way than the one built into the game, while also getting live-rendered previews of how it would look.

Turns out that that was incredibly hard 😄

[loud flint]

I feel like I’ve heard that one from several people lol

[ashen aspen]

Well, there's probably a whole "dark side" community as well that just doesn't communicate beyond their peer group..

[worthy cape]

I've definitely run into some of those people and I'm sure there's a dark mass of them, including all the grey area peeps that modify their client for noble but fundamentally forbidden reasons like performance or visibility.

The less I know about them, the better.

[ashen aspen]

It's good to have some boundaries in that regard, otherwise the game and reversing can get boring too imo.

[velvet fog]

You got corrupted, not a clean room man anymore.

[worthy cape]

Awakened.

[ornate fox]

Does anyone know the owner of the poe2planner.com domain? I would like to avoid the confusion from people that poe2planner and poeplanner are not from the same devs... seems I missed the chance to get it.

[brittle oasis]

rip, they registered it oct 29th

nothing you can really do except mention it on your website id imagine, which would just advertise it. good chance itll be pretty obvious though. yours is clean, performant and has at least a few unique features. not easily replicable

[frozen mica]

I've been working on DIY project to cluster and trend builds over the course of the league. I had it working a while ago by using the poe.ninja API to get many builds from the ladder in one call. However, that doesn't seem to be working anymore. I used to be able to send request to the poe.ninja API to gather the build data for full ladder. An example of the request script in Python from Sanctum league is below:

`ENDPOINT = f"https://poe.ninja/api/data/0/getbuildroverview"

PARAMS = {"overview": "sanctum", "type": "exp", "language": "en"}

r = requests.get(url=ENDPOINT, params=PARAMS) `

It's been a bit since I worked on this project and am trying to update the build ladder indexing script for Settlers, but that API call now returning a 404 - Not Found error. Does anyone know what the issue could be or how to fix the API call? Tagging @fickle yew just in case they have time to answer. Thanks!

[crude bane]

You have "overview": "sanctum", which I would imagine is causing you all this trouble. There might be other changes to the API since then also.

[frozen mica]

Sorry, that was just an example of a call that worked previously. Updating it to settlers has not worked.

[fickle yew]

That api doesn't exist anymore. It has been completely reworked and doesn't return all data anymore. I generally discourage people from scraping the build service - the api is much more expensive for me on my infrastructure - especially because people tend to make 100k+ requests.

However, I have started working on trends myself as well. The beginning of that can be seen on the front page now with popular class / skill combinations. If you have input for that I'm happy to get more inspiration 🙂

[frozen mica]

Got it! Thank you for the response, will save me from banging my head against a wall!

The class/skill combinations was what I was doing as well (and sometimes adding uniques into the mix if anything was interesting). Then I was measuring prevalence over time to see where the build meta was going. The visuals were kind of like your currency sparklines, but for a build's share of the meta. The project was just some python notebooks, so nothing too fancy though.

How are you doing the clustering of builds on your front page?

[fickle yew]

Right now it's still simple, just actual combinations. I do have data behind it to do sparklines as well, but didn't add them in the first version. I was thinking of making a more details page with these overviews instead of having everything on the frontpage. So it would be interesting to hear what would be great for you to see.

What did you cluster on? I did toy with that in the past as well.

[frozen mica]

I think a "meta report" could be interesting to show the top builds, trends over time, and any new combos that are popping up. I use your builds ladder to hunt around for interesting new build concepts, which is why I was building the trends to see if there were any pockets of players doing something unique. The visuals made that a bit easier.

To do the clustering, I was using unique items, active skills, and keystones for features. Then I vectorized them with term frequency and clustered with LDA. It wasn't perfect, but was good enough to be interesting. Pulling up some old data from archnemesis and an example cluster that captured the CF champ meta was: fortitude, last to fall first to strike, the gull, corrupting fever, etc. That is probably unnecessarily complicated for getting the general distribution of builds though.

[fickle yew]

Thanks. I'll play around with the data. I think it could make the build page more approachable to less experienced players too.

[frozen mica]

Looking forward to seeing what you build!

[fickle yew]

Just to clarify: Were you actually vectorizing the individual features like one hot encoding them or just combining all the features in a string before doing term frequency?

[frozen mica]

I turned each build keywordsd (unique items, active skills, and keystones) into a single string, with each keyword separated by spaces, then I featurized those strings with sklean countvectorizer or TF-IDF to get the features for each build. Then I did the modeling on that feature set.

[fickle yew]

Alright. That's how I read it. Just making sure.

[fickle yew]

It's always hard to find the subcategories in the different builds. Many popular builds like ice nova will have crit, non-crit, hybrid, CI, life and other combinations - and so it can be hard to figure out which populations qualify as a cluster.

[frozen mica]

Agreed... my best solution to that was adding the uniques in because there seemed to be some differentiation there between the variants. Adding the masteries also seemed to help with the sub-categories

[crude bane]

Been working on database and query optimization. Happy to say we have gotten the times down from 30 seconds on ~6 mill items down to consistent sub 1 second queries on ~64 mill items. Thats a combination of actually looking at the query and trying new things as well as major database changes using timescale db and indexing, but also automatic unique recognition and basetype based on modifiers present.

Very hard to stop optimizing now that we have begun and I wish I could redesign the whole system, but no way that's happening anytime soon. As it often goes, I know just how little I know about SQL, which is a great deal more than when I started this project haha.

[lone ingot]

would a daily dump of the data be plausible? I imagine something as simple as a list of all the pob links would be enough for the clever people to infer the data they need

[loud flint]

Alright it's time to get started on PoE 2 stuff

[worthy cape]

Imagine having built any system with PoE2 in mind from the start.
I wish I did 😄

[compact isle]

same

[compact wigeon]

I don't think I could have set PoP up with poe2 in mind? Not enough info

[worthy cape]

If there's anything one has learned from being around PoB for long enough, almost every league has a curveball.

For Inya, one of the largest upheavals I've had thus far was probably when the path hash algorithm changed and path reps became lowercase.

Both things I've keyed off in the database, throughout the times.

For that I ended up rehashing all historical paths, even if it's anachronistic to do so.

Still fighting the urge to do a personal v2 of Inya; it would've helped if I had deployment instructions.

[mint karma]

The infrastructure for poe.re is pretty up to date for poe2. I just need some new data sources, but I've made it so the base-data can be close to whatever, and then I just feed that in to a data-pipeline that generates typescript-files with the solved regex for frontend.

I've spent a lot of time on that over the last year. But generating a new page should take days instead of weeks now 😄

It's pretty far from perfect still, but it is what it is! I wish I had more time to work on it.

[ashen aspen]

Just looked at the generated data for poe.re on github out of curiosity.
Looks like an interesting problem to solve, finding the lowest character unique regex for an arbitrary combination of pre-defined strings while trying to make the whole thing max 50 character.

I like that map mods have a scary property. ;)

[frank drift]

it turns out 50 is not actually the limit which is nice

hopefully no limit/larger limit in 2

[worthy cape]

It is a limit, and unless you want to juggle hardware, the limit.

[frank drift]

if using/emulating a controller is "juggling hardware" sure

[worthy cape]

Is anyone's fundamental assumptions broken?

[compact isle]

Oh nice now I can talk about it

[velvet fog]

Since the Taiwan realm does not have an Epic/Xbox/PlayStation account, there is no difference. There is no news whether poe2 will be available on Epic/Xbox/PlayStation in the future.

[mint karma]

Nope. This makes a lot of sense 👍

[compact isle]

Taiwan has steam, so there is still the potential for sub-accounts and so everything (discriminators etc) will also happen for them

[arctic vale]

@civic crane Can you please add an option for Awakened to be able to use system proxy for connection similar to Telegram.
some people live in areas were ISPs blocking or having trouble connecting to the servers

[velvet fog]

I thought steam was just an account link, not sub-accounts

[compact isle]

you could theoretically have a steam-only and an email-only account that you want to merge under this system, so it has to support it

[loud flint]

I've read this through a few times, but it's not super clear to me: Does this mean that multiple of the same CHARACTER name can exist?

Or is it just ACCOUNT names that are now allowed to share names, with a discriminator behind them?

[carmine merlin]

Sounds like just account to me

[compact isle]

character names are unchanged

[carmine merlin]

Are accounts forced to have a discriminator? Will this break our saved account name list in PoB?

[loud flint]

Okay cool, one of the things that concerns me as a mirror shop manager is that things like this could lead to people being scammed without realizing, by someone having the exact same name

Also: how will it look in game in the context menu? We ask people in cases where they're unsure to check the account name, so if the discriminator shows up there it's at least some extra safety

[compact isle]

yes and yes

just imagine everyone went wild and asked support to rename their accounts to have #<4 digit number> at the end

[carmine merlin]

Will accounts by default use a discriminator of #001 unless there is an account with a duplicate name?

[compact isle]

the discriminator is random

[loud flint]

Next supporter pack: Choose your own discriminator

(Honestly, I'd pay for that)

[compact isle]

I won't rule it out for the future but currently there are no plans for that 😉

but to anyone reading feel free to ask questions. It is a complex beast of a change that has occupied much of my time for the past few months

[loud flint]

This is the one I want to know for now, basically when you right click in the chat when getting DM'ed/DMing someone (without having them as friend)

I'm not at my desktop right now so can't give a visual example

[compact isle]

it will show up anywhere an account name is, the account name has a discriminator they're not separate things

[loud flint]

Right okay, that makes it clear, ty

[carmine merlin]

I think my only potential concern is that with a random descriminator it’d be easier to impersonate another account for scam reasons. Knowing a random set of numbers is harder than just #001 for an original account owner. Not sure if this would ever actually happen where someone tries to get a descriminator that is very similar to a streamer or mirror crafter to fool someone

[loud flint]

Yeah that's sort of similar to my concern actually, since we deal with A LOT of mirror trades every League, even one of them being "middlemanned" would be disastrous for all parties involved

It's easier to tell someone that SomeGuy#001 is the real deal than SomeGuy#3485

[worthy cape]

One fun thing is even with that scheme you'll eventually have legitimate participants that have other discriminators than the one designated as OG.

[loud flint]

Valid point honestly, but at least we could keep it to a minimum

[worthy cape]

Possibly even earlier than expected if one is stuck on a weird account name and want to take the opportunity to change to their "real" one.

[loud flint]

All in all I think this is a great change (was a massive fan when Riot Games did the same), just worried about the logistics of avoiding scammers

[worthy cape]

For sure, it's a legitimate concern.

[loud flint]

But there probably isn't any fool proof way to avoid it, discriminators or not

[hazy fiber]

does this mean you could potentially email support and ask for a specific discriminator?

I assume not

[compact isle]

no, support doesn't have the ability to do that

[oak estuary]

So for looking up a character via the website, will the Xbox and Playstation realmCode values just not return any characters? Is that even a valid parameter anymore? I'm guessing there can't be a lookup available for finding account name + realm = discriminator, right?

[formal kraken]

@compact isle whats the account name returned from the oauth login api? Is it the main or sub account (aka can I login with a specific sub account)?

If it is the main account, does it include the discriminator in the id/name or separate?

If it's in the name, is it always a # character? (terrible for URLs)

Is there any migration path where the api can return me the legacy and the new name? (aka how can I associate old account bound data with the new, if I only used the account name as id)

[formal kraken]

What people on discord did was create accounts until they had a nice number...

[compact isle]

no idea what realmCode means in this context, could you clarify?

[oak estuary]

Ah, sorry it's called realm on your end. E.g. character-window/get-characters?accountName=Novynn&realm=pc

[compact isle]

@Novynn whats the account name returned

[compact isle]

this will catch you a ban pretty quick

[compact isle]

characters aren't shared in PoE1, so you still need to pass those parameters

in fact some OAuth APIs will now require them if you don't want the default, pc

[hazy fiber]

nitro let you pick numbers, but yes, you can just do that

until you make enough to get noticed and are banned

[prisma cargo]

Reporting an issue with https://poe.re/#/items regex string generator: wands do not have %inc spell damage option.

[compact wigeon]

@mint karma

[mint karma]

thanks, I'll look in to in this weekend (loads of work this week). It seems some of the data is missing.

[compact isle]

oh yeah, something to note is that item filters will be accessible across all realms now

and the realm parameter / dropdown in website is just the one it validates against

[velvet fog]

This version has many changes to dat

1. Add .datc64, .datcl64 files, which should be prepared for poe2
2. Remove all .dat files with size 12

[lone ingot]

yeah wow, loads of changes

[velvet fog]

ya...that's lots

[lone ingot]

godspeed ggg, can't imagine how much a pita this db migration is

[compact wigeon]

[velvet fog]

https://www.pathofexile.com/forum/view-post/25480464

[clever thistle]

off by five bug? 😄

[swift beacon]

Sounds like it.

[swift raven]

Not sure where to ask this so feel free to send me to another channel. But anyone ever seen this error with Awakened PoE Trade?

[lone ingot]

you don't have a documents directory?

[worthy cape]

When in doubt, blame OneDrive.

[lone ingot]

assuming it's looking in these paths and cannot find it https://github.com/SnosMe/awakened-poe-trade/blob/6a78fee1092d26817ead6611cd3575abaafd9b89/main/src/host-files/GameConfig.ts#L10-L19

[swift raven]

Yeah I just fixed it recently. It was also causing issues with my filters not being found.

I did a windows update yesterday and it either removed or changed the path to my documents folder. I had to restore it to default and lost everything in my documents folder 🙃

[lone ingot]

that'd be why, yeah, awakened poe looks in the same directory as your filters for the production_config.ini, basically a config file for poe

[worthy cape]

Microsoft likes to surprise-enable OneDrive for people and if you've done something weird to it, it can easily misbehave.

PoB's nemesis 😄

It can misbehave on its own too, but some people tinker.

[lone ingot]

I wish the windows file system wasnt so crap

[swift raven]

I don't even see one drive. I'm wondering if I was using it before and the update got rid of it along with all of my stuff xD

[lone ingot]

why did they have to commit to backslash? actual satan tbh

[worthy cape]

Hysterical raisins, be happy that most of the surface API accepts either direction.

[lone ingot]

Ive been using the rust wrappers for some of the windows APIs also and it definitely leaves something to be desired in some cases

it's swings and roundabouts I guess, wayland split in linux made things equally complicated

[compact isle]

meanwhile PHP who chose backslashes for their namespacing separators...

[lone ingot]

all the mistakes we've made with backslashes as a global community is just

[formal kraken]

\n

[lone ingot]

[prisma cargo]

@mint karma missing regex option. Belts > suffix > "reduced flask charges used"

[mint karma]

thanks, I know its missing a lot of data, I'm almost never manually entering mods, so its something wrong with my data pipeline, hopefully fixing the other issue should fix this too

[prisma cargo]

np, posting just because that's what you suggest doing in case of missing mods 😛

[mint karma]

yeah, appricate it 😄

[mint karma]

This should be fixed now.

Sometimes I dissapoint myself. I spent like an hour trying to figure out what was wrong with my data pipeline... and it turned out to be a grouping issue in frontend. The fix was 2 lines of typescript code.

[prisma cargo]

thanks 🙂

[kind swift]

Hi @mint karma found a lil mismatch here , t1 life ( "fe" in the regex) makes feathered arrow quivers match if there is no prefix to rename it ( as i guess ^fe means "starts with" fe ? the one below matches correctly cause it was a t1 proj speed and the two above dont match cause they do have a prefix that renames them

[worthy cape]

In regexes in general, ^ anchors to the start and $ to the end of the input.

[kind swift]

ty for the insight , i aint a regex expert at all , hell before using it in PoE i thought it was specific TO poe xD

then that confirms the mismatch between T1 life "fecund" and this , just changing it to "fec" i guess would fix the issue ?

[mint karma]

Thanks for the report! 👍🏻

This should be fixed now. The issue was that I didn't check the regex start and end against basetypes.

The update should fix a lot of similar issues too. You might need to click the reset button or reselect the mod for it to work.

[green rose]

is there any official documentation on how to handle existing 3rd party apps post username change? How are we supposed to associate non-discriminator usernames to discriminator usernames?

[worthy cape]

There was a thread recently with some tidbits: https://discord.com/channels/174993814845521922/1306110084861919362

[green rose]

thank you!

[lone ingot]

well done db migration people

[exotic egret]

[velvet fog]

I don't understand how to report Chinese translation problems. I wrote a forum translation report but got no response. I wrote a letter to customer service and said it would be forwarded, but three months have passed and Warden and Guardian are still translated into the same Chinese.

[exotic egret]

I passed on your message to the translation team

[frank drift]

is #0000 GGG exclusive?

[hazy fiber]

very possibly, but the _GGG and red/gold colour and Grinding Gear Games should already be exclusive to them

not that you can select the discriminator

[oak estuary]

Well the _GGG isn't exclusive, anyway 🕵️

[compact isle]

[oak estuary]

It's scammin' time

[worthy cape]

[clever thistle]

dat line-break!

[formal kraken]

Ah damn I was too slow, I played valheim on the weekend instead of preparing pobbin oops

[velvet fog]

playing Enshrounded, 2 weeks for 105 hours

[formal kraken]

I saw a little bit of enshrouded and was pleasantly surprised

Seems pretty polished and fun

[velvet fog]

Very interesting and reminds me of Minecraft days.

[carmine merlin]

@velvet fog @civic crane any idea where the ModType colomn in Mods.dat went?

[chilly oasis]

Hash16 is now 2 bytes instead of 4, so ModType is at 0xA (10)

[carmine merlin]

Thanks

[fickle yew]

@compact isle Thanks for finding all the places I didn't uri encode properly by chosing # 🙂

[hazy fiber]

its - on the website, assume to avoid needing to url encode

[fickle yew]

Yep. Maybe I should do the same for my links. But all the APIs require # as well.

[hazy fiber]

yeah it caused us (PoB) issues, becouse to get the exact account name we do somthing and that returns a - and then we have to gsub it back to url encoded #

[carmine merlin]

PoB update is out now that fixes the account import issue

[brittle oasis]

Do you guys invite only contributors to PoB discord or even people who just wanna follow your progress with poe2 coming soon?

[oak estuary]

Just contributors. Adding anyone else gets too noisy

[crude bane]

We suddenly cant access https://www.pathofexile.com/api/trade/data/change-ids anymore in our app through a get request. Even with bearer token. We don't know since how long its been happening.

Anyone else had similar issues?

[velvet fog]

$ wget https://www.pathofexile.com/api/trade/data/change-ids
$ cat change-ids
{"psapi":"2693734501-2661050503-2585239229-2872938514-2781216298","forum":"28394556"}

worked for me

[crude bane]

Seems to be an issue with the python request library, it works fine in browser and with wget

[gilded tree]

User agent?

[crude bane]

User-Agent": f"OAuth pathofmodifiers/0.1.0 (contact: {settings.OATH_ACC_TOKEN_CONTACT_EMAIL}) StrictMode"

[gilded tree]

Mh

I suppose you’re hitting the cloudflare bot page, maybe check what output you’re getting

[crude bane]

Yes, we were. My friend seems to have fixed it by just using the same user agent as the browser does and no auth token. Personally I don't feel very comfortable with the approach, not sure if it would be considered breaking ToS

[grave wren]

works fine on my side with requests:

resp=requests.get("https://www.pathofexile.com/api/trade/data/change-ids", headers={"User-Agent":"demo ([e-mail])"})

I am pretty sure the user agent with a contact email is a hard requirement

[frank drift]

yep

[lone ingot]

is this normal for the trade site or am I misremembering something https://gyazo.com/b2851c503f8efbecfa81c07e9e1c7cb9

[worthy cape]

A quick Bing suggests that "similar results" might be a "Better Path of Exile Trading" extension feature. Could you describe what the video is supposed to illustrate and how it's malfunctioning/weird?

[lone ingot]

collapse is happening on a per item level rather than account listings, for some reason I'm remembering collapses being like all of x item for y account

[compact isle]

that's definitely a third-party extension

[lone ingot]

methinks better path of exile trading has a bug then

[worthy cape]

A wild guess is that they got struck by changes around the new and improved account names and need some bugfixes to the extension.

That's the risk of anything that hooks the web, you're at the mercy of the structure and detail of the underlying site.

[lone ingot]

yeah, I just use it for the folders/trade history

[frozen wagon]

me shamelessly stealing the poe-racing UX for my private league tool:

[worthy cape]

Inspiration is flattery? 🙂

Do you do anything fancy to determine gem linkage, or is it just showing all the gems socketed in the slot?

pobbin had some fun times trying to determine how gems were linked; kind of hampered by that PoB users generally don't set up socket colours or links at all.

[strange otter]

@mint karma Believe I've found a issues with the magic mod section of regex website. Site said I could report them here.

[frank drift]

yes, you can just say what the issue is

[strange otter]

This string at the end does not work for Lich's Circlet

Also when I was crafting gloves earlier, I noticed this was matching leech. I believe both life and mana

My 'workaround' is just doing this to match open prefix, but I don't think there's a base-specific way to do open suffix?

[calm adder]

Saw this posted on reddit just now: https://www.youtube.com/watch?v=oA7xeRyxMSo

That's so cool, @fickle yew!

[worthy cape]

[calm adder]

I had no idea about the chinese poe.ninja clone Tencent spun up.
Really interesting!

[lone ingot]

I liked the talk @fickle yew, lots of good advice

the building a db part was funny

[fickle yew]

Thanks 🙂

[compact wigeon]

Still annoyed I didn't manage to attend that talk

[carmine merlin]

found out about clicking notables on the heat map from the talk. Never knew about it before

[ebon oasis]

oh i had no idea you did a talk, i know what i'm watching today

[worthy cape]

You could've just mentioned me by name in the part about scope 😛

[fickle yew]

You were obviously many times in the original script but didn't want others to feel bad..

[worthy cape]

[frank drift]

PoB trade searches are defaulting to Sony

I wonder how the list is getting sorted since it's "Sony, XBox, PC"

[carmine merlin]

was meant to show pc at the top, might have gotten messed up at some point

ahh they changed the realm names

[ashen aspen]

cool talk, nice to get some insight into the infra that runs poe.ninja

[compact wigeon]

I found it really neat to get a perspective on why not to open source and I really respect it. I've idly wondered about that but never enough to try to get an answer about it.

[worthy cape]

The other drawback tends to be "is it worth the instant ripoffs and forks that dilute your brand and make your life harder?"

It's not uncommon for tools you make yourself to be hacked by someone else to introduce a popularly requested feature that you know won't be healthy, and end up with even more requests for it and confusion among people who don't know who is behind what version.

It was somewhat prevalent around foobar2000 plugins, I had some of those.

[fickle yew]

And sometimes it's just what's needed. Pob is a great success story of that. But yeah. I want my own hacking space for this project.

[worthy cape]

PoB is impressive in how it's brought more and more contributors into the fold. It's probably helped by that it's a freestanding application and not a website or service with the potential for monetisation.

I guess the Linux/macOS versions of PoB technically count as fork forks 😄

[hazy fiber]

Are you saying I cant just put in an ad service into PoB to make it show ads?

[grave wren]

some people would claim that happened with the build list stuff we had in briefly

[worthy cape]

Only on the first day of the fourth month.

[mint karma]

What do you mean "the string at the end"?

[mint karma]

Thanks. I'll look in to why this matches leech. Which base did you use it for?

[strange otter]

The (\w+ ……., seems to be used for open prefix/suffix from my understanding; but I know it doesn’t work for some bases

Like when I did shields, the website said it would not work for titanium

[strange otter]

Warlock Gloves

[mint karma]

Ahhh, I see the issue. Haha. Regex can be a pain sometimes. Thanks for the report, not sure how to solve it in a reasonable length tho. Lich's includes ' - which isnt matched by \w

I think the real solution here would be to actually select the base as well. I'll see if I get some time this week to find some solution for it.

[strange otter]

Ya I did ^Lich to match open prefix

But I don’t know if there’s a base specific way to match open suffix too?

[mint karma]

You can do lich's circlet$ to match open suffix

clet$ is probably enough

[strange otter]

Oh thanks that helps a lot, I should be able to make that fit

Fantastic website btw, thanks so much for developing it

[frank drift]

“I know, I'll use regular expressions.”   Now they have two problems.```

[fickle yew]

That is actually one that is somewhat mitigated by llms. They're very good at not only writing but also reasoning about regex.

[exotic egret]

"Lacking manpower"

[ebon oasis]

that was a really nice talk rasmuskl :)

[ebon oasis]

no shit, my first idea for doing harvestVendor was: yo, it will be amusing to ocr the text and then regex the shit out of it...

i did change it to something that was less convoluted once ppl actually were interested in it :D

[tawny merlin]

Thank you @fickle yew for sharing your experience and the good spirit behind :), really nice to hear you talking about what you faced in this project.

[fickle yew]

@ebon oasis @tawny merlin Thanks! Appreciated!

[formal kraken]

I really have an urge to build something like poe.ninja just because it's a fun technical challenge, the reason I am not doing it is, because I know I won't ever be as good as poe.ninja without a significant amount of time invested and even then I'd probably just end up not being better.

[fickle yew]

But pobb.in is awesome. Did you run out of ideas for it?

[formal kraken]

<3, no just in a cycle where I want to do everything else

[fickle yew]

I understand that feeling completely.

[formal kraken]

But basically that was just me trying to say, what you've built is awesome and I'd probably never reach that

[fickle yew]

Thanks. It did take 1000s of hours literally though. But has been a very nice thing as my job became less about writing code myself.

[velvet fog]

zao cong

[formal kraken]

oh they accepted the red names 😮

[hazy fiber]

no more hiding it I guess, zao is now red

[fickle yew]

Yeah public gratz @worthy cape

[worthy cape]

[formal kraken]

I feel that, escaping into my own code is so nice after a meeting/management rich week(s) at work

[ebon oasis]

you know what tool i miss ? something that could mostly automatically (if possible) track all my trades

[hazy fiber]

cant do that easily sadly

best you could do is guess from .txt

[ebon oasis]

yeah i assumed as much from what i know

[worthy cape]

The big bugbear there is Kakao (South Korea) accounts that can't be logged, which makes any trade assistant a bit spotty.

[grave wren]

just hook into windows ai search and hope your trades happen during the screenshot 😛

[ebon oasis]

look into the log txt for trade start and activate a screenshot to trigger when you click in the area of the accept button...

now that i wrote that out, that would actually be somewhat doable

[gilded tree]

Or setup your own recall (e.g. https://github.com/arkohut/pensieve, there are multiple projects) and use Ollama for local visual search. But you definitely need a trigger or tiny interval.

[ashen aspen]

if you always reply "thanks" (or something more unique) to the person you traded with, you could match that up with the trade request in the log

it's not foolproof, but screenshots aren't either

[ebon oasis]

the important would be to detect the opening of the trade window, everything after that isnt an issue

[gilded tree]

could lead to some false positives if you press cancel instead of accept

[ashen aspen]

idk, you'd still need to somehow "mark" a trade as completed..

too many edge-cases

[ebon oasis]

the stupid way is to just define the coords of the accept trade button is on my screen, and listen to mouse click in that area

[ashen aspen]

poe should give out notarized invoices for each trade ;)

[ebon oasis]

i agree :)

[frozen wagon]

not yet, I noticed some x/y properties on the items but too busy with my normal day job to look until later 😦

[dense shore]

smh ggg poaching 3rd party devs again

[frozen wagon]

@fickle yew I'm so impressed you got PoB running in docker and working so nicely with your stack, I don't suppose you published an image to docker hub, or would be willing to provide some advice for how I could achieve something similar?

Loved the talk btw, very inspiring to me for some of my own side projects.

[prisma cargo]

@mint karma hey, it's me again 😄 Regex for magic items misses 2 categories: Minion wands, Minion Rings

[oak estuary]

PoB can be run in headless mode, you just need LuaJIT. Rasmus does some custom setup, but check out HeadlessWrapper.lua in PoB. It's the same kind of thing we use for unit tests

[tranquil token]

@velvet fog "Recently" menu showing the recently added MTX has disappeared from the main page, so could you please add it again? link address is remains. https://poedb.tw/us/MicrotransactionRecently

[velvet fog]

Because v3.25.2 recently moved all MTX to new dat files, there is no way to compare the records of previous versions.

[tranquil token]

Oh, I understand, thank you for your response.

[fickle yew]

Yep as Wires mentioned I'm just using HeadlessWrapper.

[mint karma]

I'll generate it when I get some time ^^

[velvet fog]

HeadlessWrapper is hard

[frozen wagon]

Thanks a lot

[loud flint]

@fickle yew consent popup seems suuuper bugged right now on poe.ninja, I get prompted on every page

[fickle yew]

Thanks. Taking a look.

[fickle yew]

Anything special for your setup? Using an unusual browser or other factors?

[loud flint]

I just realized I landed on the site with appended ., so poe.ninja., so let's just chalk it up to user error

FWIW you should probably rewrite poe.ninja. to poe.ninja

[fickle yew]

Oh wow. That's a funky error.

[loud flint]

URLs are inherently funky 🥲

[golden bane]

You've asked about this before: #tooldev-general message

Have you tried it? Did it not work out for you?

[frozen wagon]

I did try but didn't get super far with luajit + docker, I will try again, sorry.

[rose quail]

hi, sorry if this is the wrong channel, but is there a way to download crafting modifiers/modifier groups/weights from poedb or some other such site? in like json format or whatever

i can also hit an API if that's easier- i just need a one-time download of such crafting info for now

(also, dang, rasmuskl is here! i just watched their video from CPH devfest- huuuuuuge fan!)

[frank drift]

would suggest starting with RePoE https://repoe-fork.github.io/ https://github.com/repoe-fork/repoe-fork.github.io

[rose quail]

ah ok, cool, thanks! ah ok, i can probably read in these json files and take out just the stuff i need. probably a good idea to have an explicit external data->internal app representation conversion process anyway. thanks!!!

[worthy cape]

The poe.ninja appreciation club meets on Friday afternoons, right after force pushing to prod. 😉

[grave wren]

UTC?

[worthy cape]

One of those :30 or :45 time zones.

[grave wren]

[rose quail]

10/10 i love it. and by "it" i mean force pushing to prod on fridays, especially right before a holiday during a code freeze. it's my favourite activity of the year

[crude bane]

Seems like the issue was the Authorization field, whenever it was included it would result in a 403 error. Not sure if that's intended or not, perhaps someone more knowledgeable of API design can explain?

[compact isle]

OAuth tokens give you less permissions than a guest, as they're restricted in scope to their specific permissions

[crude bane]

Ah ok, when put that way it makes a lot of sense

[frozen wagon]

@golden bane thanks for the motivation, I managed to get an alpine image working with luajit/PoB after reading this:

https://github.com/Openarl/PathOfBuilding/issues/28

FROM akorn/luajit:2.1-alpine

RUN apk add --no-cache git

RUN git clone https://github.com/PathOfBuildingCommunity/PathOfBuilding.git /app

ENV LUA_PATH="/app/runtime/lua/?.lua;/app/runtime/lua/?/init.lua;;"

WORKDIR /app/src

CMD ["lua", "HeadlessWrapper.lua"]

Now to work out how to parse builds to your xml format and then build a server on top of it.

[hexed mesa]

are there any limits in use of the psapi/river api? I maybe could use it for my bachelor's thesis

[worthy cape]

Restrictions like for commercial purposes? Good question.

[hexed mesa]

well, i dont think my bachelors thesis would be commercial in any way (as far as i understand the word anyway lol).
mostly thinking about sharing the data with my teachers if i needed to or possibly making it public overall.
realistically speaking i would replace the account names with some self made IDs and other than that idk if theres anything, but then again, maybe its just beyond the scope of the api purposes and would be promptly rejected with "dont do that"

dont really see anything within the docs/tos though, and psapi used to be publicly accessible without oauth in the past as well. asking just in case though

[worthy cape]

Do you still have an OAuth application by the way? Registration of new ones was inhibited last I looked at the dev docs.

[hexed mesa]

yes I still do have my oauth app, and unironically what I was considering for my thesis would be pretty much what I wanted to do with that app as well (although in much more limited scope)

wouldn't be surprised if registrations were reopened before I actually start doing anything though. not really sure about the thesis timeline but I think it's still a couple of months before we start actually doing anything

[ashen aspen]

Been patiently waiting for oauth applications to open up for months now ;) I don't have a lot of hope.

[worthy cape]

From a personal perspective, it sounds like a fun thing to do. I'm generally a bit reluctant to mix that kind of fun idea with school projects as they may not always be feasible to do and get a good academic outcome out of.

[hexed mesa]

oh yeah, im absolutely aware of the risk involved with using it for school, for all we know psapi could be scrapped between me starting the thesis and finishing it, or simply change drastically for one reason or another, which would be somewhat not great i suppose

its more of a rough idea for now anyway, will have to pitch it to my supervisor and see what they think, maybe they wont like it at all

mostly im worried about it being okay or not to use the api in this way, hence the original question

[grave wren]

All bachelor projects i saw weren't requiring that much in terms of being a viable product. Was mostly to showcase the student's abilities.

for all we know psapi could be scrapped between me starting the thesis and finishing it, or simply change drastically for one reason or another, which would be somewhat not great i suppose
Even if the api was gone you could still keep the thesis project running by just caching a few pages and doing your work with the data assuming ggg doesn't mind it.

[compact isle]

PoE tool development discussion. NOT IN GAME DEBUGGING. Not how to use tools. :stackoverflow: :readthedocs:

[lone ingot]

anything interesting in the 4gb update?

[worthy cape]

itemvisualheldbodymodeloverridebyitemaffiliatedattributes rolls right off the tongue... I thought grantedeffectskillsetsperlevel was bad 😄

[hazy fiber]

poe2 tree when /s

doesnt seem too hard to implement for tooldev tbh, so you guys have another 3ish weeks to polish it before you "need" to give it to us

[brittle oasis]

Did they say they'd release tree data at start? I always figured they'd want players to spend more time in game or something

[hazy fiber]

They have not said anything, and they might not release it directly, but I assume that it might be available on the website, and if you it can easily be scraped anyway

[brittle oasis]

Ah duh, good point

[frank drift]

the old fashioned way

[oak estuary]

How does this work when approaching the length limit of a username? I don't know what the limit is to test, but will the discriminator always be # + 4 digits, or can that get truncated to fewer digits at the extreme end?

I'm curious if any existing usernames were at that limit and you just bumped the DB column size to get around it 😄

[compact isle]

migrating from UTF32 to UTF8 helped a lot in terms of space 😄

current limit is <32 without discriminators

but there will be accounts that are longer than that

[vivid compass]

Yo ! From what I saw they plan on making poe2 auction house for items too right ?
Is that planned to happen in poe1 ?
I kinda want to make an auto seller, for personnal challenge, but if auction house its irrelevant right ?

[hexed mesa]

I don't know what you exactly mean by auto seller but sounds against ToS

[vivid compass]

Oof
Idk it seemed challenging like recieve message
Invite
Do math to find the place where the item is
Trade the item

Maybe it’s against tos
But making Poe tools sounds fun I wonder how their api is

[grave wren]

trading alone would be an issue when doing it automatically

you're not really allowed to automate much

[vivid compass]

That makes a lot of sense yeah

I will find an idea for something to do, maybe another copy of exilence next, challenging the API and data possibilities?

[grave wren]

lots of possibilities yeah and always more fun when it is for stuff you care about

[vivid compass]

The API doc isn’t too bad as well will try something out :D

[hexed mesa]

you are out of luck because they dont accept new app registrations for now

[hexed mesa]

and according to hst its been like this for months, so might be a while longer

[crude bane]

Any ideas to remove these kind of outliers?

Some guy has listed his 1 passive voices for 666 Mirrors, and sometimes he is the only one online with a 1 passive. The rest of the prices are "only" in the 1-2 mirror range. Even if I stop collecting duplicates, there would still be the occasional "showcase" prices slipping in that would make it look ugly. Probably gonna go over to daily data points, but then keep the last 24 hours as per hour (so I still need to remove these outliers somehow)

Also, as a sidenote, should I connect two data points that are more than 1 hour apart?

[frank drift]

I would probably start by just setting a "reasonable" bound for prices, like 10 mirror, and just ignore any data points above that

beyond that, you probably want to trim data points, possibly Winsorize trim

[crude bane]

Such a simple idea that I had not even considered, many thanks. Will look into the winsorize trim

[crude bane]

Seems like we were already doing this, just not the squeeing. But that doesn't help when there is only one data point for that hour. Would need something more fitted for Delta analysis

[frank drift]

I was assuming you're taking a range of say, 24 h (though you probably want to set the time range based on the volume) when deciding whether a point is an outlier

very hard to draw valid inferences when there's one/none on the market at any given time anyway

[half urchin]

hi. i have this app I use to score stash rares, and I can't log in and retrieve the stash json anymore. its in python.

https://www.pathofexile.com/character-window/get-stash-items?league=SSF Necro Settlers &tabs=0&tabIndex=2&accountName=AccName_1234

This was it and it doesnt work anymore. Please let me know if you got a clue

[frank drift]

it would probably be educational if you attempted to figure it out on your own first

[crystal pawn]

Not sure where to ask this but will GGG be releasing an API for POE2 somewhere in the Early Access window?

[frank drift]

there has been no announcement about that, just have to wait

[crude bane]

Endt up using the winsor trim to find the 95th percentile of all the data points and then using that to set the upper y-axis limit. So no manipulation of actual prices, but outliers are visually excluded excpt for connecting lines. However, if you hover over empty regions where outliers are, it still shows up on the tooltip.

Feels like an ok solution. It felt wrong to use it per data point, as usually outliers only occur when its the only one listed, so the displayed value in those cases wouldnt reflect anything useful at all.

[crude bane]

Finally somewhat happy with the website. Gonna try and somehow hitch on to the current POE2 hype. Here is a link to a reddit post talking about the tool:
https://www.reddit.com/r/pathofexile/comments/1gy79ok/path_of_modifiers_10_make_educated_trades/

Want to be able to iron out the minor/big bumps before next league

[frank drift]

neat

I'm a little confused about requiring an account, is caching not good enough because of the potential number of combinations?

[crude bane]

Potential number of combinations? Neither of us know much about how the internet works unfortunately

So we just went with what seemed like it would work. Personally I'm not a huge fan of it, but there is some sunken cost fallacy in there too.

[frank drift]

I saw you have Cloudflare, I assume you're caching queries and having Cloudflare serve them

probably requiring people to make an account is going to be a large barrier to getting users, even I am not personally interested enough to make an account

if you really want to tie people to a signin, you might want to do a PoE signin instead

password based signin is being abandoned pretty much everywhere

[crude bane]

Shouldn't be too hard to rip out the account feature, it's been turned off for most of the development

[charred lake]

Hi, I came across this info about the _.index.bin bundle file https://github.com/poe-tool-dev/ggpk.discussion/wiki/Bundle-scheme#bundle-index-format
Just wondering if this is still valid? I just took a look at my own index file with this schema and the first u32 (bundle_count) has a value bigger than the number of bytes in the entire file, so couldn't be right 😅
It does mention that "The binary index file is a compressed bundle", but doesn't seem to play ball with an off the shelf oodle decompressor I found.
Any pointer greatly appreciated!

[vivid compass]

fr ? damn....
You mean i cant get like tokens i need to request the API ?

[civic crane]

yes, valid.

[charred lake]

Ah figured this out, the file has to first be interpreted as a bundle, then the contents of that bundle are decoded and interpreted as the index

[knotty falcon]

is there a place I can ask about PoB? I have an issue, and cant find the right place to ask...

[hazy fiber]

github issues is the best place, but you can ask here if you cant ask there, or DM one of the PoB devs

[knotty falcon]

nm, account retrieval changed in the latest update, and my account name for GGG side was wrong

was missing #XXXX

its....been a while 🙂

[crude bane]

I understand what you mean now I think. Was too hung up on finding a method that can't be abused for assigning users a unique token that's not reliant on creating an account with verified email.

We are not caching any queries because a) there are so many combinations, but maybe the more popular ones could be cached (we are tracking incoming queries), b) we want to serve the freshest data available, meaning caching is only useful (most of the time) for an hour and then it must be cleared, and lastly c) we haven't thought of a good way to really do it.

We do want to cache basic get requests for modifiers and base type information, but haven't gotten around to it. All this is very overwhelming when you have no expirience and no seniors to guide you away from the worst traps.

[loud flint]

UUID session token?

Depending on backend that might already be baked in

[ashen aspen]

Requiring users to sign up using their email address opens a whole can of worms, you're suddenly (legally) responsible to keep this data save and make sure that it doesn't leak in one way or another. A lot of people (rightfully so) don't trust you to keep their data save.

[frank drift]

poeninja down, how rare

[worthy cape]

worksforme™️

[ashen aspen]

One trap you're describing could be to go overboard with features that are inherently unnecessary to the problem you're actually trying to solve. Examples for this would be an account system, a public API, splitting a small scope application in micro services and generally just "overengineering".
This can lead to loosing focus on what the app is actually about, putting too much time into things that users don't even see or care about and not enough time optimising what users do care about (i.e. usability).
It can also lead to a lot of overhead when it comes to maintaining the application and can make adding new features very slow and hard.

Of course it's your app and you decide what direction you want to go in and that's totally fine. I'm sure you learned a lot when making this and if that's the goal, you likely succeeded!

Sorry if this is unsolicited advice, but it sounded to me like you were looking for some input.

[frank drift]

very short downtime I guess

[crude bane]

It is yes, and we already have ip rate limiting implemented for some of the endpoints. Just gotta do some QA and it will be ready

[crude bane]

Your advice is appreciated. Overengineering has been a concern of mine, but when one of the main goals of the project is to learn new things, it has become hard to balance simplicity vs. learning. Its what happens when you have no outside testers that can offer fresh input. We will focus more on usability, since people actually using the website would be a good source of inspiration.

[frank drift]

if you're on cloud hosting, double check your budgets and alerts

[ashen aspen]

https://poe-racing.com/
unfortunate to see the site being discontinued

[dense shore]

man thats a piece of history disappearing

[crude bane]

#tooldev-general message
No more accounts needed 😄

[frank drift]

Grats. Only necro settlers?

[crude bane]

Yup, although I don't know why we don't bother with regular settlers. Too busy thinking about other stuff I guess

[frank drift]

Settlers would probably be more interesting as something to compare next league to, since Necro Settlers economy is extra wacky

[pallid wraith]

Well, since you can manually migrate out of necro settlers to Settlers, its economy will also be more wack

[ivory epoch]

How we contact Chuan for info thats not on poe2db

[pallid wraith]

Normally, the majority of poedb is populated via datamining

Until he can properly datamine poe2, poe2db can't really get much data

[ivory epoch]

He doesnt have the ritual or delirium atlas nodes, numbers for detonate dead, corrupting cry or scavenged plating.

[frank drift]

are you saying you want to provide it, or complaining it's not there

[ivory epoch]

Stuff i found so far that i search and dont find on poe2db

[frank drift]

I don't think he would mind if you @ him if you have data

[crude bane]

Its so late in the economy, so we focused on necro settlers. But I guess, the lanterns make it equally or more messed up

[velvet fog]

thanks

[restive jay]

[loud flint]

all hands on deck for PoE 2 I'd bet

which as a sidenote

holy shit i'm excited

[dry sky]

Hola!
Is there any API for POE2 available?

[compact isle]

none immediately for early access, but we'll add what we can during it

[dry sky]

Looking forward for making crossplatform build configurator

[mint karma]

Looking at the brightside, the upside of no APIs for the start of early access is that all the tooldevs can play the game! 😂

[hazy fiber]

"all"

[gilded tree]

all api connoisseurs

[mint karma]

Yeah, "all" indeed. There is also a lot of stuff you can prepare without APIs. I'm for sure ignoring all my responsibilities, both IRL and otherwise, for the first five days though!

[frozen wagon]

@worthy cape I finally got around to doing the gem sockets + links with some calculations (damn this took some fancy css grid + absolute positioning over the top to get right):

[worthy cape]

Ooh, fancy.

[frozen wagon]

Ty, it's all open source too (https://github.com/hevans90/smol-app/tree/main/src) but relies on a pretty custom set of typescript/react utils and a tailwind config. If anyone wants to make something similar on the web I am more than happy to share a minimal setup for doing this from https://www.pathofexile.com/character-window/get-items

[dense shore]

Does the tailwind make it work 8% faster

[gilded tree]

tailwind 4.0 will

[charred lake]

Hi all, I've built a set of CLI tools to pull out the "real" files from the Steam bundle files. https://github.com/adamthedash/poe_data_tools
Hopefully it'll allow people who are new to tool dev (like myself) to get over that initial hurdle of digging into the files to see what's there 🙂
I've also included some diagrams of the bundle file format as I couldn't find a lot of easily digestible info around.
For now now it just supports linux, since one of my dependencies is only built for linux right now, but hoping to support windows soon.

[worthy cape]

Pretty diagrams, I used to have that font as my main IDE font in the past.

[loud flint]

That definitely looks like Excalidraw!

[wild geyser]

lmao thats insane

[waxen musk]

it seems visualggpk2 doesnt work anymore, and 3 doesnt have table view yet? what tool can be used to preview tables?

[charred lake]

https://snosme.github.io/poe-dat-viewer/ this is the one I'm using atm

[velvet fog]

why keystone in passive skill tree have only out without in?

        "11239": {
            "skill": 11239,
            "name": "Wind Dancer",
            "icon": "Art/2DArt/SkillIcons/passives/WindDancer.png",
            "isKeystone": true,
            "group": 604,
            "orbit": 0,
            "orbitIndex": 0,
            "out": [
                "39718"
            ],
            "in": [],
        },

[civic crane]

uint16 for hash was added recently, maybe it's the reason

[brittle oasis]

I remember there being more keystones that are the same way, even regular nodes that dont follow the same patterns. Just gotta consolidate them into a single connections list and do your own thing afaik

[hazy fiber]

11239 < 39718, connected nodes that are larger are supposed to link via "out"

it just makes it easy to find direction and makes checking if its connected faster, but most of the time its easier to just combine in and out and treat it as one big connected list

[daring moss]

Has anyone come up to take over now that poe-racing.com is going down?

Been meaning to mess around with some web stuff again so I thought I might as well implement something (even if never meant to go into "production") but if someone else has already done so I may as well try to contribute there

[lost cobalt]

hello all, trying some tooldev for the first time but am a bit confused on how to find image assets; for example in https://github.com/grindinggear/skilltree-export/blob/master/data.json I find every passive node, and they have icons like Art/2DArt/SkillIcons/passives/plusintelligence.png, but there's no similarly structured folder... where is this referring to?

[brittle oasis]

The json has sprites.imageName.zoomLevel.coords.iconName

[lost cobalt]

am I missing something? https://raw.githubusercontent.com/grindinggear/skilltree-export/refs/heads/master/data.json has no references to sprites at all, only icons?

[lost cobalt]

oh sorry my bad, the json hadn't loaded properly... I see them now. thanks!

is there a similar resource for items like currency and skill gems and base types etc.?

[brittle oasis]

https://github.com/repoe-fork/repoe-fork.github.io
https://repoe-fork.github.io/ to be exact

[worthy cape]

That's a stale RePoE, the current one is lvlvllvlvllvlvl's.

i.e. https://repoe-fork.github.io/ https://github.com/repoe-fork/repoe-fork.github.io

[brittle oasis]

What zao said

[lost cobalt]

ok I see it has info like this;

"visual_identity": {
      "dds_file": "Art/2DItems/Maps/AtlasMaps/FragmentChimera.dds",
      "id": "ShaperFragment3"
    },

but where's the dds files?

[worthy cape]

There's an Art tree on the RePoE website with PNG/WebP transcodes, but it's ultimately sourced from the game bundles.

[lost cobalt]

ahh I see, thanks! looks like I can use the dds_file path appended to https://repoe-fork.github.io/ to get an image

[pastel marten]

I built a replacement with dummy API responses but the process for requesting API access has been closed for months so I can't launch it

[velvet fog]

https://www.pathofexile.com/forum/view-thread/3591631

[vapid pulsar]

@worthy cape or @compact isle
Are you able to comment on the PoE 2 skill tree export?
Will it be in the same/similar format as it is in PoE 1?

[worthy cape]

No info from me.

[compact isle]

I wouldn't expect any exports until some time into early access

[carmine merlin]

Be back to yoinking from the website then I’m guessing?

[compact isle]

I mean that's the problem, there won't be any tree on the website for launch

too much to do unfortunately

[waxen musk]

whats the difference between dat64 and datc64?

theyre identical as far as i can tell?

[carmine merlin]

No differences that I’ve found. Think they might be dropping regular 64 at some point so I moved to using c64 for now

[crisp kestrel]

https://www.pathofexile.com/forum/view-thread/3592012

When i click on each spoiler in this thread first time, it delays for 150-200ms(Chrome browser).
Checked in Firefox - no delay.
Asked friends - they experience the same. Any ideas?

[ashen aspen]

collapsible elements still suck in html. ggg seems to handle this in pure js which triggers a lot of layout repaints. i'd guess it doesn't help that the images inside are dynamically sized based on viewport and are huge in file size.

[calm adder]

How do you see that diagram?

I assume it's a feature of chrome dev tools?

[crisp kestrel]

yep, this is Performance tab

[calm adder]

Ah I see

Interesting view

[mint karma]

https://poe2.re/ this will buy me some weeks to start implementing some things hopefully 😂, and I'm hoping poe2 doesnt need as much regex as poe

[hazy wigeon]

Wanted to poke my head in here to query on if folks were fiddling with a pob2?

[mint karma]

I kinda want to contribute to pob2, I just hope it isnt going to be written in lua 🔥

[oak estuary]

We still need to make an official announcement, but this is the basics: https://old.reddit.com/r/PathOfExileBuilds/comments/1h2b758/path_of_building_2/lziusjd/

Probably will be an initial Lua version, since a rewrite will take time and we don't want to commit too hard in one direction until we know how the game will be played. There's progress on a Go version happening here: https://github.com/Vilsol/go-pob

[frank drift]

Vilsol of course

[bright ridge]

probably old news but just noticed the new 'Game' selector on the trade site 👀

[formal kraken]

could have given a dev a heart attack if you quickly edited the PoE 1 to PoE 2

[compact isle]

That's been there a while :p

[hazy wigeon]

Does it solve the "can't run in the background when not focused" problem pob1 had/has?

[wind jolt]

Hello there guys! I have a question, I'm a software dev and as a hobby I wanna get into developing tools for poe2, what should I start with to get used to it faster? Any ideas for a quick base project to get the base knowledge down?

I've been looking through this text-channel for a while for now.

[hazy fiber]

depends what kinda tools you want to work on, most of the interesting tools wont be very useful at the start of poe2 becouse lack of apis

[hazy fiber]

and this ^ is PoBs response, so unlikly to be able to help with that

[wind jolt]

Alrighty, thank you for answering!

[clever thistle]

personally i consider this a feature 🙂

[oak estuary]

Tbh, it was added as a feature, so I do too, even if there was a better way

[green rose]

I assume we won't have the currency exchange API during the beginning part of EA?

[primal pewter]

There is a currency exchange (not just trade) API confirmed?

[exotic egret]

No currency exchange API is confirmed

For clarity, there's no implicit comma after "No"

For any of you sneaky grammar people

[pallid wraith]

(us sneaky grammar people wouldn't assume you were making a mistake)

[jolly spear]

Guys, do we know if PoE 2 data files will be in the same format as PoE 1? I have an app idea but I'd need to dump the items' names from the game files for that or get it from an API and I don't know if poe-dat-viewer would work while we don't have the API

[frank drift]

wait and see

[vapid path]

Hello i suddenly get this "path of exile abnormal disconnection failed to join any instances" Why?

[dense shore]

This isnt tech support channel

[carmine merlin]

They might use the new datc64 type which just seems to be a renamed dat64 but will have to wait until launch to know for sure

[charred lake]

Oh cool! Good excuse to learn some Go and do some tinkering!

[swift beacon]

Not sure if anyone else has noticed this, but in the POE1 files it looks like "four" might be used as an alias for POE2. There are some files/assets that include the word "four" in their paths and appear to be from POE2.

For example, there's a 3D model in a folder named "charactersfour" called marauder.fmt, which looks like it might be the POE2 model for the Marauder (it's much nicer than the POE1 model). There's also some assets with paths that include "act1_four," "act2_four," "four_act4," etc. Some of them don't match the corresponding POE1 acts, such as the "stonefigure" models under act2_four, and so might be POE2 assets.

There's not much, certainly no big spoilers that I've found, just a couple of things I found interesting.

[hexed mesa]

makes sense given that poe2 used to be poe 4.0

good old days when it was just an update to poe1

[worthy cape]

I petition to rename them to poefourmaster.

[modest path]

Program Files (x86)

What year is it

I assume the standalone client has everything encrypted?

https://patch-poe2.poecdn.com/4.1.0.4/Bundles2/Content/*

from an error installing in wine lol

[hollow vapor]

Hey guys, I'm trying to download the POE 2 client, but when I start the exe and downloading begins, it says http 403

What does that mean?

[modest path]

http 403 is permission denied error

it most likely means they are having issues, unless you have some sort of proxy or av software that's messing stuff up locally

Yeah I get the same thing right now

Though it worked a few minutes ago

[golden knot]

This is a channel for tool development, not technical support. This would be better posted in #2┃early-access

[modest path]

I tried pointing poe-dat-viewer at the new url where the poe1 bundle would be but it's not there

structure change maybe

[hazy fiber]

they dont include dats in preloads

atleast not normaly

[modest path]

Ahh

As long as goblin troupe art and audio is still in the ggpk

I'll be ok

[idle dome]

Can the content.ggpk be read with the existing tools for poe1?

[oak estuary]

Likely

[modest path]

Bundles2/Folders/1E/data.datc64.bundle.bin referenced in download log

[velvet fog]

dats in preloads

4.1.0.4 -> 4.1.0.5

[hazy fiber]

oh intersting

[modest path]

hmm, why they make it an encrypted prerelease on steam if it's just out and about on the standalone

[oak estuary]

Because that's how preloads work on Steam

[idle dome]

You can read about how preloads work here if you're interested: https://steamdb.info/blog/steam-download-system/

[lone ingot]

yeah, preloads are encrypted by default on steam

has anyone grokked the poe2 dat file?

[worthy cape]

Greetings, fellow exiles. How are you doing?</meme>

[modest path]

I like the implication that it's been a meme until now

[raw otter]

nice bait

[oak estuary]

Ha, that's great 👏

[swift beacon]

Interesting that they're keeping the 4.x version numbering for POE 2. I wonder if that means POE will never get a 4.0 major update and forever be 3.x.

[modest path]

they do have it prefixed as pathofexile2 a lot

[lone ingot]

what tools are people using? Im still waiting on the download with my slow australian internet 🙃

[rain jacinth]

PoE2 is shaping up to dwarf PoE, also on launch the version number they present is 0.1.0, they are using 4.x internally - which makes sense given the history

[modest path]

I can't imagine them not going skelecrew on poe1 pretty quickly if poe2 doesn't bomb

poe1 going to have tf2 spinbots in a year

[rain jacinth]

I think they will give a bit more resources into PoE1 as long as it has players. Many PoE1 players might bounce off PoE2.

[modest path]

I mean it doesn't take much to outdo half of one of valve's interns

[hazy fiber]

it could skip to 5.0

but generally the X.0 updates are restricted for large campaign tweaks

[waxen musk]

There's a predownload (non steam)?

[frank drift]

yep

[waxen musk]

ah yea i thought it was poe1 or something

wait no it is poe1 😐

ah its a new website i see

[idle dome]

what viewer is that?

[modest path]

poe-dat-viewer - I assume on poe1 data 'cause the schemas are different for the poe2 files

speaking of, is there any coordination on poe2-dat-schema?

poe-dat-viewer happily opens the datc64s if you rename them to dat64, but opening mods.datc64 only sees id and hash16

[carmine merlin]

I just downloaded the game files and got it set up for me in PoB. A lot of the dats seems to be similar

[waxen musk]

wanted to ask that too, i have a lot of changes in the schema but idk where to push them

[modest path]

Is @civic crane active or is creating a repo a reasonable step

I am not a JS guy and have... Unpredictable motivation, so I am probably a bad repo owner

Could just fork poe-dat-viewer with the intention of making it support both I guess

erm, poe-dat-schema

names

well both kinda I guess

[waxen musk]

theyre active yea, i assume theyll do it when they wake up or something

[modest path]

I have been convinced to actually try to get a plan together re playing or whatever

[strong pulsar]

psg files seem to be quite a bit different, may be missing something obvious though

[waxen musk]

they added a new int for connection radius

[modest path]

In what context?

[waxen musk]

and a new int for each root node but idk what that one is for (theyre all 0 i think)

wym what context, just every connection in psg is now node id + radius

[modest path]

oh, missed the passive bit - Interesting

[waxen musk]

so each connection can be made into an arc instead of line

[strong pulsar]

thx @waxen musk, that helps a lot

[zenith zodiac]

does anyone know the socket command to get the patch server url? (e.g. for poe1 it's echo -en '\01\06\x1a' | nc patch.pathofexile.com 12995)

[nova hamlet]

@velvet fog mod weight info is just gone from the client dataset? Assumed server-side now?

[strong pulsar]

@zenith zodiac echo -en '\01\07' | nc patch.pathofexile2.com 13060

[waxen musk]

oh yea seems so. thats a bummer, cant automatically figure out which mods go on items now 😄

[frank drift]

they hinted they were doing something like that

[waxen musk]

i get wanting to hide actual weights, but it also means we dont know which tags allow which mods 😦

[carmine merlin]

Yeah it messes with a lot of things for 3rd party tools

[modest path]

spicy

[carmine merlin]

maybe it's just left out of the preload?

[frank drift]

I wouldn't be optimistic

[waxen musk]

preload is pretty much complete data otherwise

[primal pewter]

Didn't GGG at some point say that even for PoE1 they didn't intend for the spawn weights to be in the client data and it was a mistake they ended up in there?

[waxen musk]

yea that makes sense. a lot of games do that. but it would be nice if we at least had the tags 😄

[lone ingot]

that's kinda cool in a way, we can get t0 mods now

u can roll ilv2 boots for explode mod at 1/1000000 chance, prove me wrong

[loud flint]

wait that's annoying

i'm gonna be honest, i think this is a mistake if this is the plan moving forward

[strong pulsar]

how would one figure out orbitRadii & skillsPerOrbit without the skilltree export? Is it in any .dat64 files?

[lone ingot]

the cdn is getting absolutely smashed

[worthy cape]

I had pretty much wire speed on both Standalone and Steam now tonight.

Did some region wake up? 😄

[lone ingot]

unless this is a just misleading error I am seeing, I dont remember it when I was looking at stuff on the patch serv before

Service Unavailable
The server is temporarily unable to service your request. Please try again later.
Reference blah

https://errors.edgesuite.net/blah

maybe I am just hitting something that doesnt exist and getting this, idk

[gilded tree]

EU 😄

[waxen musk]

open the texture with lines and measure them 😄

as for skillsPerOrbit its two more 72s and a 144, you can kind of guess it by looking at all nodes using those radii and taking a max of their positions

[rain jacinth]

This only changes things early on each league. They will just need to get estimated. Shouldn't be a problem to get estimates tight enough with help of friends.

[loud flint]

...Which is what we've had to do with Synthesis implicits until now, and whatever weight lists we have of that is definitely not 100% accurate yet

IMO as a crafter, I don't like it at all

[waxen musk]

well, games have the right to withhold that information, leaving it to community to figure out. just like you cant datamine vendor recipes

but it sucks that you cant even build a list of potential mods now

i went from like 10 possible prefixes for an armor (excluding all the league mechanics and influences) to 100+

[rain jacinth]

More people will be bothered by this and willing to contribute in an organized manner. Synthesis implicits are bit of a niche thing and isn't the mod pool of them relatively large?

[loud flint]

https://synth.pathoftools.com/?baseType=Helmets relatively big and diverse yeah

The problem is that community contributions will never reach the exact measures we had before

We will approximate it close ish with enough contributions, but we will never be 100%

[carmine merlin]

Anyone have their schema for fixing the gem dats?

[rain jacinth]

I guess lack of alts/chaos does make data collection much more annoying and laborious.

[lone ingot]

no snosme lib no dat files for me, never bothered to implement myself

[civic crane]

working on it 🙂

[lone ingot]

haha I just saw, I was modifying it for datc and repulled and got confused

thanks, appreciate your work

[waxen musk]

i have schemas for just about everything that matters

@civic crane can i upload the gqls somewhere?

[lone ingot]

https://github.com/poe-tool-dev/dat-schema

[simple ravine]

I'm baack 🙂

[lone ingot]

PR to that link probably @waxen musk , snosme has write access to it

[waxen musk]

but is it in the same repo?

[worthy cape]

Contrary to popular belief, there will be more PoE1 leagues. 😉 Did you intend to link the PoE1 repo?

[waxen musk]

exactly

theres still quite a bit of room between 3.25 and 3.99 😂

dont think overwriting poe1 schemas is the play

[simple ravine]

With semver, you can have 3.101 as well 🙂

[waxen musk]

was the first patch after 3.0 called 3.1?

[worthy cape]

Not sure if you were around for the 3.9 -> 3.10 patch, but it was just as amusing as you'd expect.

[waxen musk]

ah yea

[simple ravine]

I was. I was here when it was 2.4 🙂

[worthy cape]

Some exiles believing it would be followed by 4.0, others being confused why it was 3.1 again.

[simple ravine]

When shaper was new

[civic crane]

wait bit, gonna create folder, export and all that stuff

[waxen musk]

trailing zeroes

[simple ravine]

I wish the day had more than 24 hours, me having energy like a 25 year-old again, so I can dust off my ggpg/dat stuff and dig into this

[lone ingot]

I assumed poe2 stuff will cont in same repo in a branch or something

[worthy cape]

Second game, second breath.

[waxen musk]

how long ago was 3.0 now?

[lone ingot]

is there another repo already? does that make sense?

[waxen musk]

why not

[simple ravine]

Anyone thought of using openai o1 model to dissect the dat stuff, to find patterns?

[waxen musk]

o_o
dont think you need ai for that, you can just use normal logic to guess what is what

[simple ravine]

@civic crane needs to implement this in his dat stuff, which is better and more complete than mine

[waxen musk]

they follow pretty specific rules, for instance arrays of foreign rows always use increasing pointer values

[simple ravine]

There has been a lot of guessing over the years though

[waxen musk]

well yea i mean you can write code to do the guessing if you really wanted to

doesnt need to use actual machine learning

[simple ravine]

I haven't seen the contents of it yet, but guessing there will be some wildly different things in there

[lone ingot]

there's a decent amount that's untyped in the poe1 schema but everything of value seems like it's been typed

[waxen musk]

its not that much actually

some tables got messed up more than others, but mostly its just a few values here and there

the hardest part is figuring out what table the ids point to

[lone ingot]

does anyone know the answer to this?

[waxen musk]

what is the question? whether theres a poe2 dat schema repo? not yet it seems

let the man cook

[simple ravine]

I would have imagined a fresh take on the game engine, and everything, when they created PoE 2, and the inner workings. Considering that over extended period of development time, when looking back, you often have epiphanies of "if we would have done it again, we would have made it differently"

[waxen musk]

but then theres "if it aint broke, dont change it"

[lone ingot]

no, I was just wondering if there was direction on it. I got asked why I linked to the poe1 repo and it just confused me lol

[simple ravine]

dont' we have poe-tool-devs org for stuff like this, which we created back in the day

[lone ingot]

you dont need a new repo in git, you can just do branches, it's very common for that kind of thing. the repo owners can just choose to merge your work into a new branch

[waxen musk]

new repo probably makes more sense than a branch since theyre not exactly connected anymore, unless we believe some future changes to poe1 dats will be reflected in poe2 as well?

[civic crane]

@simple ravine that TODO depends on you 😳

[lone ingot]

intuitively I dont think that ggg will come up with fundamentally different data structures, in my mind it makes sense for them to be adjacent

[simple ravine]

I have some issues to resolve there 😄

[lone ingot]

in the end I dont mind that much though, just explaining the confusion

[waxen musk]

the question is, do the changes need to interact with each other? if they will be completely separate from now on then it doesnt matter

[lone ingot]

my thinking was more that the structure is there, we are just talking about the output that changes. all a new repo does is (Im assuming here) clone the structure to change the output, when what we care about is changing the output

that said it would need to be represented as a diff package and it isnt a monorepo atm, so it would need work anyway

just thinking about consumption for apps that support 1 & 2, less moving pieces the better

[carmine merlin]

@waxen musk can you let me know when you push your schema somewhere so I can have a look at it

[waxen musk]

well i can push it anywhere if theres a repo, but i also put new types kinda randomly in core.gql so someone might not be too happy with that

[lone ingot]

branch will be fine, Im assuming the next few days will be messy for tool devs haha

[jolly spear]

Oh, no way to get info yet from the files? 😭

[waxen musk]

i dont think the dats have a version anywhere, it usually came from the server 😄

[lone ingot]

better to get things working first

[civic crane]

you can type 4.1.0.5 there

[lone ingot]

4.1.0.6 is current I think

[jolly spear]

Oh, damn.

[waxen musk]

oh, straight past 4.0?

[lone ingot]

this gives current latest I think?

anyone know the cdn url for images/art etc?

[zenith zodiac]

yes. l updated this repo to output it as json https://lvlvllvlvllvlvl.github.io/poecdn-bundle-index/poe2/urls.json

[waxen musk]

ok i made a fork https://github.com/d07RiV/dat-schema-poe2

[carmine merlin]

Thanks

[nova hamlet]

I agree.

[pure shard]

@velvet fog Hey.
Just wondering if there's any way we could keep the tooltips in the gems again on Poe2db?
Like we did in the previous versions.

I used to be able to search by tool tips for specific interactions, but now I see that it fetches data on hover - and stops me from searching for tooltips things.
(Is this fetching more costly now as well?)

As an example:

• I'm trying to find all gems which reference the Armour Break mechanic.
• Unfortunately only gems with "Armour" in the name show up.

[waxen musk]

im guessing level/stat requirements on poe2db werent updated either

[nova hamlet]

idk levels and tags look fine on poe2db, its only the weight info that looks like it was axed

I mean, modifiers-wise

[simple ravine]

How's it all going guys with the dat files and trees?

[carmine merlin]

dat files are mostly figured out with some new key columns being mapped to somewhere we're not sure yet

not sure on the tree stuff, none of the PoB devs have worked on it yet

[rancid tinsel]

hi guys, is there a c# lib to parse the .datxxx files, don't wanna create an unnecessary duplicate if there is already a maintained library

[modest path]

@rancid tinsel not sure specifically about c#, but most tooling is in python/js/lua with some movement on rust and a bit on golang - in my experience stuff outside of that is mostly projects people start then abandon

[rancid tinsel]

yeah, ppl do that when they don't have enough motivation

[modest path]

@rancid tinsel if you're looking for fast results, poe-dat-viewer has a js tool that will fetch the dat files and convert them to json based on poe-dat-schema https://github.com/SnosMe/poe-dat-viewer/tree/master/lib

If you're looking to have fun writing a c# file reader, have at it raw :)

[rancid tinsel]

thank you, I'll probably go ahead and create a c# lib, based on SnosMe's and https://github.com/aianlinb/VisualGGPK2/tree/master/LibDat2 which seems outdated

would work nicely with https://github.com/aianlinb/LibGGPK3

unless @vestal pond is already working on one

[vestal pond]

Yeah, that library is no longer maintained.

[rancid tinsel]

are you planning to create a new one or should I go ahead and start one?

[charred lake]

Are the schemas for the dat files all crowd sourced? Or is there something with basic structures like column data types in the game files themselves

[vestal pond]

Are there any features that it currently has trouble achieving? (Except the DatDefinition.json, I gave up on maintaining it)

[modest path]

@pseudo ocean there's a pretty limited number of types in the dat files; the main unknown is the column names

and where cross-table links point

[oak salmon]

I don’t have a link on hand but I’m pretty sure one of the devs of poe overlay community fork has a C# dat extractor that he keeps up to date

[modest path]

and values for enum columns

[rancid tinsel]

haven't tested yet

[rancid tinsel]

will try to find it, thank you

[modest path]

hmmm, is patch.pathofexile2.com:12995 down?

I swear that worked before for getting the current version

[oak estuary]

Someone posted this above: #tooldev-general message

[strong pulsar]

was working last night atleast

[modest path]

oh I'm on the wrong port, it's on 13060

[charred lake]

Gotcha yeah, but bar snosme's (& now riv's) schemas there's no source of truth that can be ripped out?

[oak salmon]

I think this is the one https://github.com/WhiteFang5/PoE-Asset-Updater

[modest path]

There have been some column names exposed in game dev talk videos, but the ones that matter for most tool dev fall out pretty clearly

And the community column names are different

ggg folk have even said in here that some of their column names that they use internally are wrong

so if you're going for max truth you have to pick community truth or ggg truth

and to get ggg truth you have to go work for ggg

[oak estuary]

I must've missed that message, that's funny

[modest path]

If you're after accuracy and want to reimplement stuff natively in a language, I think the lowest it is sane to go is to depend on poe-dat-schema

we ended up using jsonnet to generate rust data structures from poe-dat-schema that are structs with proper pointers to other tables; that worked pretty well

and is technically pure rust after code generation ;)

Currently it's loading the json from pathofexile-dat into those data structures at runtime currently, but there's been some movement on native rust tooling at a low level since I was working on that (had a motivation change break)

[rancid tinsel]

my entire tooling is written with c#, with a lot of automated code generation in the background, so I'm actually trying to create something which works with that, using schemas, the json / etc generation doesn't seem to be that complicated, references are up to the user I guess

also, a nice learning opportunity 😊

[modest path]

Fun for sure

[modest path]

Is this changing?

I have a hexdump of 13060 working in my terminal but it's not now, and i swear the original one was different

[simple ravine]

I made one, but it's old now (.net 6) and not super complete, but feel free to use it as inspiration

https://github.com/andreandersen/PoeSharp

There's also https://github.com/andreandersen/PoeSharp/tree/fix/new-dat-spec, I need to figure out why I did not merge that one. I think it got into a whole re-architecture IIRC

[modest path]

oh no it's just going up and down lol

[strong pulsar]

has anyone put together any passive tree exports? I had finished positions last night before I went to sleep, but computing the node description text looks to be a bit involved

[civic crane]

I'd say poe-tool-dev/dat-schema and the lua one inside pob are most maintained

also what #wiki guys are using?

[waxen musk]

🎄

[strong pulsar]

masteries are disabled right now right?

[waxen musk]

yea ignore them

i think they kept them in the data because some nodes show mastery background, but the stats shouldnt be a thing

[loud flint]

Best of luck every GGG employee today

Shit's gonna be on fire

[oak estuary]

Masteries have always been in the data for this reason

[waxen musk]

well they had selectable stats

[oak estuary]

You have a repo for this? Other psg parsers I know of are out of date

[oak estuary]

I mean always as in even years ago, like pre 3.0

[waxen musk]

not public no, but its just two new int32s

one for root nodes (always 0) and one for connections (radius)

[modest path]

I have a super fast app up with both versions at http://poe-versions.obsoleet.org/ - https is being slow provisioning, I think I have CORS set correctly

No caching atm, it's making requests to ggg for every request. Will add that in a few

[strong pulsar]

https://gist.github.com/qcrist/3078c2bbc55401d911583819a65e8bf9 has psg new structure + my terrible java code

[modest path]

@civic crane https is working now

[spiral pasture]

So what are we running atm?

[simple ravine]

Now that I think of it, I am getting an itch to revisit PoeSharp library.

Wonder if third time's the charm in a re-architecture of it 😄

[gilded tree]

I feel like now is a good time for that 😄

[simple ravine]

Yeah. To be honest, my interest was diminishing over time, because PoE 2 was "around the corner", and didn't know if it was going to be a complete overhaul filesystem wise or not.

And of course, life... But hey. Now we know, and that's good!

[modest path]

Didn't you hear? Life is suspended in 40 minutes

[simple ravine]

Indeed, I'm in queue.

[modest path]

why hte hell is my steam not letting me download yet

erm

install

[simple ravine]

Let's find out if my 1080 Ti can handle this. Waiting for 50-series before ordering new parts.

[modest path]

Best GPU ever

My EVGA (rip) is still in my media machine

[simple ravine]

Was looking at the 4090, but meh. I'll wait a bit more. Need new cpu etc as well. Still rocking a 7700K. Will proably go AMD next, so waiting for that 9950X3D

Want the extra vram for 3d rendering stuff in Houdini/Octane

[modest path]

Swapped to AMD myself (7900XTX)

[fickle yew]

I only asked 800 times 😄

[fickle yew]

How are you liking it?

[sturdy fern]

seconding rasmukl's question i just bought one and am excited to have it delivered

[pure fossil]

Hey all, is there a need for node connections? ie "id": "K1", "connections": ["S60"]

[simple ravine]

❤️ Life came inbetween

[fickle yew]

It happens. You still need to come by Copenhagen and have a beer at some point 🙂

[simple ravine]

Yes!

[worthy cape]

I know @final onyx did some .NET EF shenanigans too, but not sure if they've kept up with the times.

[hushed relic]

@velvet fog thak you so much for all the poe2db work, it's so helpful!

@fickle yew @simple ravine awesome to see you guys chat, you've always been examples of awesome devs to me :o

[simple ravine]

Hey NS, how are you doing? Long time!

[hushed relic]

Bacame a solution architect, working on game projects, stuck in login screen (as everyone else)

how about you? :)

[simple ravine]

Congrats man!! Been changing jobs a bit, as well. Did move from Sogeti to Nordcloud, became a CTO there for a global service line, and then switched, because.. reasons. Now at a smaller place. Not sure what I want to do with my life 😄

And yes, login screen!

[austere fossil]

yes thank you all

[simple ravine]

You work with game development, or as a side project?

[final onyx]

can confirm; i wrote my tooling in C#, and haven't been working much on it recently
and whilst i'm happy to share anything I've learned, I have not open-sourced my code

[rancid tinsel]

inspired

[simple ravine]

Did u get my stuff to run? Damn!

[compact wigeon]

Impressive!

[rancid tinsel]

using Newtonsoft.Json.Linq;
using PoE2.DatFiles.SchemaTypes.v5.Models;
using System;
using System.Text;

namespace PoE2.DatFiles.SchemaTypes.v5;
internal class DatFileParser : IDatFileParser
{
    private static readonly byte[] DataSeparator = [0xBB, 0xBB, 0xBB, 0xBB, 0xBB, 0xBB, 0xBB, 0xBB];
    private static readonly byte[] StringNullTerminator = [0, 0, 0, 0];

    public JArray Parse(ReadOnlySpan<byte> data, object schemaTable)
    {

        if (schemaTable is not SchemaTable tableDefinition) throw new ArgumentException("Invalid schema table type.", nameof(schemaTable));

        var table = new JArray();

        var rowCount = BitConverter.ToInt32(data[..4]);
        const int dataStartIndex = 4;

        var stringTableOffset = data.IndexOf(DataSeparator);
        var dataEndIndex = stringTableOffset == -1 ? data.Length : stringTableOffset;
        var dataSection = data.Slice(stringTableOffset);
        var rowLength = (dataEndIndex - dataStartIndex) / rowCount;


        for (var i = 0; i < rowCount; i++)
        {
            var row = new JObject();
            var rowStartIndex = dataStartIndex + (i * rowLength);
            var rowData = data.Slice(rowStartIndex, rowLength);
            var currentOffset = 0;
            for (var columnIndex = 0; columnIndex < tableDefinition.Columns.Count; columnIndex++)
            {
                var column = tableDefinition.Columns[columnIndex];
                var columnName = column.Name;
                if (columnName == null)
                {
                    columnName = $"Unknown{columnIndex:000}";
                }
                switch (column.Type)
                {
                    case "bool":
                        row[columnName] = BitConverter.ToBoolean(rowData[currentOffset..]);
                        currentOffset++;
                        break;
                    case "string":
                        var offset = BitConverter.ToInt32(rowData[currentOffset..]);
                        var strBuffer = dataSection[offset..];
                        var strLen = strBuffer.IndexOf(StringNullTerminator);
                        if (strLen % 2 != 0)
                            strLen++;
                        row[columnName] = Encoding.Unicode.GetString(strBuffer[..strLen]);
                        currentOffset += 8;
                        break;
                    case "i16":
                        row[columnName] = BitConverter.ToInt16(rowData[currentOffset..]);
                        currentOffset+= 2;
                        break;
                    case "i32":
                        row[columnName] = BitConverter.ToInt32(rowData[currentOffset..]);
                        currentOffset += 4;
                        break;
                    case "f32":
                        row[columnName] = BitConverter.ToSingle(rowData[currentOffset..]);
                        currentOffset += 4;
                        break;
                    case "array":
                        currentOffset += 16;
                        break;
                    case "row":
                        row[columnName] = new JObject();
                        row[columnName]["table"] = column.References.Table;
                        row[columnName]["row"] = BitConverter.ToInt32(rowData[currentOffset..]);
                        currentOffset += 8;
                        break;
                    case "foreignrow":
                        currentOffset += 16;
                        break;
                    case "enumrow":
                        currentOffset += 4;
                        break;
                    default:
                        throw new NotSupportedException($"Unsupported column type '{column.Type}'.");
                }
            }
            table.Add(row);
        }
        return table;
    }
}

not clean at all

[simple ravine]

DId u try the fix branch or main branch?

[rancid tinsel]

fix branch

[simple ravine]

Could probably do some looking-into-this-things during the christmas holidays to kickstart something there 🙂

[rancid tinsel]

gonna work a bit more then push

[simple ravine]

Wanted to try to see if source-generated dat-file-readers

based on schema, should make it ultra fast, .NET 9 / AOT + source gen

Curious on some use-cases as well

Like, for companion apps or?