#tooldev-general

1 messages Β· Page 139 of 1

pseudo ocean
#

Rust is cool

mortal bone
#

interesting πŸ˜„

buoyant dagger
#

did something change with the API that "lastActive" info is no longer shown when retrieving characters in an account?

#

it no longer does it since the most recent update

timber path
#

I noticed a similar thing when trying to obtain the stash tab names, you now have to include a "tabIndex"

civic crane
#

they added pinning to UI, pretty sure touched other code as well πŸ˜„

unknown.png
timber path
#

I noticed get-characters now also includes a pinnable value for each character

civic crane
#

also it seems get account by char name API is removed

timber path
buoyant dagger
#

but as far as i can tell it also broke the ability to switch your character on the top left of the website

#

and lastactive is no longer there either

timber path
buoyant dagger
#

yep

civic crane
#

and I just wanted to introduce it in awakened too πŸ˜„

buoyant dagger
#

hopefully it's just an oversight considering that the ability to switch characters on the website is also broken

timber path
#

On the website you can pin characters, but not unpin them πŸ™ƒ

buoyant dagger
#

i can unpin

#

but i can't switch the character on the top left

#

there were other changes too, it's no longer possible to link directly to someone's atlas passive tree

timber path
buoyant dagger
#

in your account landing page

#

there's a button to unpin

timber path
#

Ah ic now

buoyant dagger
#

this used to work

#

now it only works when i'm logged in to my own account

timber path
timber path
buoyant dagger
#

that would still require people to change it any time they go on a new character

#

it's how the old trade macro used to work, wasn't great

timber path
buoyant dagger
#

i guess if that doesn't come back we can just match the characters array against the last few lines from client.txt

slim pilot
#

Hey everyone, I am trying to re-build my old shitty as fuck python tool in Go, but I still dont know how to hide OAuth 2 key while still showing the code to users. Like I don't know store key in server and authentication from there something. Can anyone guide me?

worthy cape
#

That's an interesting repo name, and also not public.

violet path
#

3400 model is pretty advanced, you're right to ask for help

worthy cape
#

Typically in a service you would load secrets from the environment or files, data which you do not commit to the repository.

civic crane
worthy cape
#

And yes, you need a distinct server hosted somewhere secure away from your clients to do the whole OAuth thing vs. GGG. You can have that code public, but you need to have the secrets secret.

slim pilot
slim pilot
ornate fox
buoyant dagger
#

No, i made a post on the forums but haven't gotten s response yet

buoyant dagger
#

Considering the loss of functionality like switching your profile character on the top left of the website, I'm inclined to think some of it might be a bug

ornate fox
buoyant dagger
#

Retrieving last active? Yeah it's pretty neat

buoyant dagger
ornate fox
buoyant dagger
#

Oh right, derp. It's early. That one is anyone's guess KEKWLaughing

worthy cape
#

Did anyone look if there's any change in the privacy knobs, used to be grouped under one of the other permissions before?

#

Saved some streamers a fair bit of bot updating when people incessantly asked about their atlas tree.

ornate fox
#

are you suggesting there is a checkbox for atlas trees now? (because i dont see it)

hazy fiber
#

hes suggesting if there isnt, they should add it

buoyant dagger
#

I checked and it's not there, it's just broken

#

I tried unchecking everything and it still didn't work

tawny agate
#

BTW, I don’t know who is in charge of the Twitch extension in GGG but I have a suggestion: add in the thing a link to the public profile

worthy cape
#

The Armoury extension?

tawny agate
#

Yep

slim pilot
vapid pulsar
grave wren
#

is there a way to use this as an enduser already?

vapid pulsar
#

this just tells you the replacement of a passive node by it's Id

grave wren
#

thought so but wasnt sure, good work anyway πŸ™‚

late schooner
vapid pulsar
#

I have got a rough version in web assembly on a tree but needs a lot more work

unknown.png
slim pilot
vapid pulsar
#

@carmine merlin did you get a response from GGG about this jewel stuff?

carmine merlin
#

Yeah just for PoB not to be the first ones to post the info

vapid pulsar
#

Ok so sounds like they wont be going after us to take things down then?

carmine merlin
#

Not if what Bex told me remains the case

copper steeple
late schooner
#

will there be a giant json of every jewel and every notable? for the lazy peoples

carmine merlin
#

Someone will probs create one

grave wren
#

Time to release an XML Flaskium

velvet fog
#

You can select Unique jewel, seed, Basic Jewel Socket and it will display all passives inside radius.

Tool page: https://poedb.tw/us/TimelessJewel

poedb
PoEDB, Path of Exile Wiki

PoEDB provides things come out each league, as well as items, uniques, skills and passives. Path of Exile Wiki editing functions.

copper steeple
#

@velvet fog Tried to send you a message, but I think you have receiving messages from strangers disabled. Can you accept the friend request?

odd sail
#

How big is the data set to iterate over every seed in every socket for a specific unique jewel and conqueror?

#

I assume capable people will quickly crank out a reverse DB search and find the seeds that result in high quantity of specific jewel modifiers in opportune socket ids and snipe them off of trade

copper steeple
#

I hope so. πŸ˜›

odd sail
#

I mean… your tool with two extra for loops can already create the DB…

hazy fiber
timber path
patent notch
#

running Victario right now [nvm just saw conq type doesnt matter i think]

#

funny thing is i didnt find any Miniondmg x4 in elegant hubris caspiro, maybe someone can correct me if i am wrong

#

[in templar area using these easy to access nodes "Overcharge", "Endurance", "Faith and steel", "Holy dominion", "light of divinity", "divine fervour", "devotion", "divine fury", "divine wrath", "arcane capacitor"]

timber path
patent notch
#

i see, must be on the other node then i mentioned

timber path
#

Your list of templar notables is still missing some notables from what i can see

patent notch
#

Rites of Solaris

timber path
# patent notch Rites of Solaris

That isn't part of the normal tree though? I've pm'ed you a list of notables I'm using (don't want to go too much off-topic in the tools channel here)

patent notch
#

oh tysm

hazy fiber
#

if anyone wants to make a proper search tool for things like weighted sums of timeless jewels, I recommend using these binary precomputed files, description of what they are, how to access data, and suggested method of searching is in the readme

if theres any errors please inform me, and I might put Glorious Vanity in here later if anyone asks for it, but atleast not in the next 16ish hours

πŸ“Ž JewelData.rar
fallow zinc
formal kraken
fallow zinc
#

that's great but as far as I know that's still windows only

formal kraken
#

works on my Linux Β―_(ツ)_/Β―

hazy fiber
hazy fiber
fallow zinc
#

nice

hazy fiber
#

updated readme formatting, @fallow zinc let me know if you endup doing anything interesting with it

fallow zinc
#

I'm assuming it's C# or something, I don't know anything about that language and the files are compiled so I doubt I will

#

if the data were another format maybe I could mess around with a language I'm familiar with

formal kraken
#

the data is independent of the language, how to use it is in the readme

fallow zinc
#

still need to load the data though

#

it has no extension, and it's not plaintext, so I don't really know what to do with it

civic crane
#

need to add .bin , then people will start looking for docs (README) πŸ˜„

late schooner
#

sounds like a riddle. any simple way to convert this data to json?

hazy fiber
#

its literally just binary data, each byte is a value of the array

#

should have made that more clear in the readme

fallow zinc
#

I never work with binary data, everything you just said is still a mystery to me I'll be honest

late schooner
#

that part is clear, but the converting method not really

hazy fiber
#

just convert each byte to a short short (uint8) and stick it in an array, its setup for fast access atm, it takes me around 6 seconds to do full complex weighted sum search on all 84 jewel variations (4 jewel types * 21 sockets, not including Glorious Vanity)

#

including time to load files into memory

formal kraken
#

@hazy fiber why did you serialize it to hex, you could mmap the whole file at once if it wasnt

hazy fiber
worthy cape
#

GH may attempt to visualise it poorly.

formal kraken
#

oh it just happens to look like it

worthy cape
#

Seems suspicious tho, everything is in printable text range.

formal kraken
#

oh yeah doesnt even look like hex, I should have actually checked

hazy fiber
#

some of them look wonky, and some look like pure gibberish

hazy fiber
# worthy cape

jewels fall into a range of values, it happens to line up like that, look at Militant Faith for example

worthy cape
#

Ah, of course I looked at the file that was in the happenstance range.

#

I'm probably a bit damaged by previous text work, immediate feeling I got from this was: "Russian?"

unknown.png
formal kraken
#

that's awesome

worthy cape
#

Python:

lut = pathlib.Path('Militant Faith').read_bytes()

C++:

std::ifstream is("Militant Faith", std::ios::binary);
auto file_size = is.seekg(0, std::ios::end); // or use std::filesystem::file_size on a path
is.seekg(0, std::ios::beg);
std::vector<uint8_t> lut(file_size);
is.read((char*)lut.data(), lut.size());

Probably JS, I don't speak it:

var req = new Request(".../Militant%20Faith");
fetch(req).then(function(resp) {
    return resp.arrayBuffer();
}).then(function(arr) {
    var buf = new Uint8Array(arr);
});
#

Can't actually test JS because it's the worst platform in existence and CORS is apparently enabled on everything I run πŸ˜„

#

@hazy fiber I think one of the hold-ups for people might be the term "convert", rather than read/slurp/load as-is.

hazy fiber
hazy fiber
fallow zinc
#

Ill try a variation of that python example in ruby

#

I was on a whole different track

fallow zinc
hazy fiber
#

I might put my example and zaos python/c++ code in the readme if @worthy cape is fine with it, along with any wording/phrasing advice he can give me

worthy cape
#

(I don't know Ruby πŸ˜› )

#

Take anything you want.

fallow zinc
#

I'm pretty sure the above snippet just reads the 42nd character of the file

worthy cape
#

The goal is to have an array with each of the bytes of the file, without interpreting it as text or some other encoding.

#

The raw byte value in some offset is what you wish to consider as an 8-bit unsigned integer.

#

Note the 'rb' to read it as binary rather than text.

fallow zinc
#

oh damn I missed this if no change is made to the node then index is 249 -- my previous code might have worked

hazy fiber
#

I am just glad I did not include my current iteration of Glorious Vanity, its way more complex

first it includes all the small nodes so the array is much larger
secondly it uses index, value pairs becouse it has roll range, then it can have up to 7 different pairs per node per seed, to make it seekable without a ton of deadspace it has lookup table at the start (considered doing just size and then having to manually create the location, instead of giving location and having to use it to determine size) the lookup table are uint16s and the rest of the array are uint8
so basically to get a node you have to do

uint16 array = loadArray()
uint8 start* = &array[indexSize + array[node_id_INDEX * jewel_seed_Size + jewel_seed_offset]]
uint8 end* = &array[indexSize + array[node_id_INDEX * jewel_seed_Size + jewel_seed_offset + 1]] + 1
#

and then you have to make a list of changes and values from that

patent notch
hazy fiber
# patent notch https://github.com/KeshHere/TimelessJewelData Just in case anyone is interested,...

its nice, but why xlsx and not csv?, 2 why as there are 543 Notable skills theres only 392 on the tree (list here https://github.com/Regisle/TimelessJewelData/blob/main/node_indices.csv I am guessing this includes ascendancies?)
and 3, more useful elsewhere, as a dataset like the one in the github I linked is more useful for fast access (pure binary file <3 MB, instead of yours at 16MB) for tools, where yours is nice for human readability, and lastly please put <> around your links

patent notch
hazy fiber
patent notch
late schooner
#

will you repeat for other jewels?

patent notch
#

Yes I will

hazy fiber
patent notch
#

I see, well I wasn't sure how many total are there so went with highest number in a sense i guess.

trim lotus
#

Hey guys, is there any source for a list of passive points that are in range of any jewel socket?

late schooner
#

on poedb

hazy fiber
#

if thats what hes after

late schooner
#

yeah

trim lotus
#

This is a nice start, but yes, would love all the small ones aswell πŸ˜„

trim lotus
patent notch
hazy fiber
#

They dont count as being in range of jewel sockets, those are pre computed what nodes are in range, and ascendancies are excluded

#

and dont think theres code in the ggpk to exclude numbers, as the prng function is used elsewhere iirc

#

@worthy cape updated my readme

formal kraken
#

does it actually have to be a uint8 or could you compress the data into let's say 4 bits?

hazy fiber
hazy fiber
# formal kraken does it actually have to be a uint8 or could you compress the data into let's sa...

you could, but it makes it slower
you know the jewels dont overlap so you could setup custom values for output to correspond to a specific outcome, I think you can only reduce it down to 5-7 bits depending on which jewel, its not a big deal to do, as the slowdown would be in loading it in, and not searching, you can also then run a compression algorithm on it, because its similar values, some of them have low compression ratios but its an average of 55% (becouse I think elegant hubris is like 70% becouse of all its no-op)

#

assume compression ratio would be significantly lower if you reduced the output size

formal kraken
#

put yeah probably doesn't matter much with a good compression

hazy fiber
formal kraken
#

and in memory size is not that relevant

formal kraken
#

but also more curisoity

hazy fiber
#

you defs wont get the vaal jewel down to any reasonable size, you could throw away alot of the info, like small nodes/values but it would still be atleast 3 times the size of any of the others

#

maybe it would compress nicely though

late schooner
fallow zinc
#

I think I'm close but I probably missed a detail

hazy fiber
patent notch
fallow zinc
#

ohhhh it's offset

#

hmm no that still doesn't make sense

hazy fiber
#

I may have encoded this one incorrectly, also got 249 for it and adjacent nodes, hmmm

fallow zinc
#

do you think it could be a mistake with this one file? or possibly all of them

#

I can do another type of jewel real quick to sanity check

hazy fiber
fallow zinc
#

ok let me try one of those

#

for the index in those files, should I be keeping 0 as the headers?

#

or remove the headers

#

or am I cooked and am I thinking of another file

hazy fiber
#

not sure what you mean, heres a py script to just grab a bunch, results should be 40, 40, 41, 41

Nodes = [191, 76, 128, 93]
seed = 16141
seedOffset = (seed - 10000)
seedSize = (18000 - 10000) + 1

for i in Nodes:
    print(lut[int(i * seedSize + seedOffset)])
fallow zinc
#

ok that's helpful, at least that way I'll know that part is correct

hazy fiber
#

you had a few errors, jewel seed size is +1, (7901 possibilities for Elegant Hubris), jewel_seed should be jewel_seed _offset, my initial readme incorrectly stated it as just seed so my bad on that one, otherwise you had correct method

#

I think mine might be shifted, will look at it again in a bit

fallow zinc
#

sure thing

hazy fiber
#

sorta busy atm sadly

fallow zinc
#

ah no worries πŸ™‚ I'll work on lethal pride for a bit

hazy fiber
#

hence slow response

grizzled seal
#

I saw two reports on reddit of jewels being offset, I think it's a bug in the original implementation

#

neither user clarified what jewel or seed, they both reported the same offset issue though

slim pilot
#

oh my fucking fuckery fuck

#

every valuable seed is getting sniped

hazy fiber
worthy cape
slim pilot
#

I seen a str stack seed but not anything else

patent notch
hazy fiber
#

the nice thing is it makes it public

grizzled seal
#

they already were being sniped for years now, it's just widely known instead of limited to like 100 people

slim pilot
#

yeah and that too.

fallow zinc
#

does anyone have an exampel of a lethal pride or brutal restraint location, along with the additions or changes people want on them and which specific nodes?

#

I wanna try building a search for them

hazy fiber
#

most interesting ones are searched, if you just want some practice, grab ones with multiple str % or %dex nodes respectively in radius

#

or if you wanna look for meme ones, look for somthing like warcry effect, or fortify effect

slim pilot
#

is there a public search engine tho

hazy fiber
worthy cape
#

Exercise left for the reader.

slim pilot
#

well omw to write an utterly slow search program then

velvet fog
#

I will make one later tonight or tomorrow

slim pilot
#

nvm learnt it

late schooner
#

545

slim pilot
#

thnx

hazy fiber
#

do note, it seems my Elegant Hubris is wrong, ran through the others with no issues so far

#

but gives you an idea how to store em if you want to do it yourself

velvet fog
#

Generated and stored in sql

hazy fiber
# velvet fog Generated and stored in sql

kk, not a bad way of doing it,

mine was setup to optimise for speed (can do a full weighted search on all 84 jewel possibilities (4 jewel types * 21 sockets) in about 6 seconds including time to initialise search program and load data into memory)
only takes up like 12MB uncompressed, missing Glorious Vanity becouse it has a bunch of other things you need to worry about

I would assume SQL would still be fast and small enough for poeDB

fallow zinc
#

@hazy fiber I checked my code against what you wrote above (the example giving 40 / 40 / 41 / 41) and I get the same result

#

however when looking up what change that is, I get a different result from poedb

hazy fiber
#

maybe my seed is offset by 1 then?
I just ran a check and it gave me same result using the LUT and by rnging it from scratch

fallow zinc
#

what's LUT ?

hazy fiber
#

look up table

fallow zinc
#

oh

hazy fiber
#

ran through 64000 random values in both lethal pride and brutal restraint and all returned same result

hazy fiber
fallow zinc
#

thats not the one I put into pob, let me check tho

#

for this one poedb lines up

#

what the heck

hazy fiber
#

@fallow zinc millitant faith seems wrong as well, Ill have to go back and look at them again, if you can give me a pair that doesnt line up with poeDB ill try and see if its an error on my side

fallow zinc
#

I think I got it... probably got confused and set the wrong id or seed somehow

velvet fog
#

I only use the random generator, the rest of the process is written in php, if there is a problem, it may be compared with other generators

hazy wigeon
# velvet fog Generated and stored in sql

This will definitely be the way, this is the exact type of use case for sql queries to shine, wont be long before we just have a simple database you can download and query against to min max

fallow zinc
#

my code seems to work

hazy wigeon
#

And if a weighted search is pretty fast, PoB can integrate with that (note that you dont have to actually check against which notable gives which for the search, you merely need to just know how much dps each stat gives and then weighted search those, which notable they hit doesnt matter)

fallow zinc
#

8k jewel seeds, looking for about 8 passive additions on about 8 specific passives... takes about a second to run

hazy wigeon
#

you have it up on github?

fallow zinc
#

not yet, its too manual

worthy cape
#

Days since disappointment with LMDB: 0

#

Incremental database file growth on Windows still not in a release despite a patch being merged to master in 2015, thanks to the author really really not liking Windows.

fallow zinc
#

@hazy fiber any leads on the lethal pride stuff? or is it possible I still made a mistake there

#

(or basically any of the data files that dont work right now)

hazy fiber
#

lethal pride works, same wit brutal restraint, the other 2 seem wrong, I am about to go to sleep but I will fix them tomorrow

#

I have an idea why they are wrong

vapid pulsar
#

Lol i love GGGs naming of the scourge UI Hellscape Microwave

unknown.png
fallow zinc
#

ohhh right I confused lethal pride with another one

hazy fiber
#

the dataset atleast useful for the 2 which work?

slim pilot
#

so uh I did a search engine kinda but works with only one notable. then why the fuck i wrote this

worthy cape
#

Education! πŸ˜›

slim pilot
#

lmao

#

have to try at least two more notables

#

still learning go...

worthy cape
#

(really, that's one of the best reasons to do stuff, always something to learn)

#

Still highly professional, I see.

slim pilot
#

uh thumbnail

polar wren
#

got my gui done to search for specific mod combinations on specific notables

mental edge
polar wren
hazy fiber
#

@patent notch interested in speed of your search (that you just posted to reddit/github), compared to using the binary data setup I provided
(a search on a single jewel socket with 21 notables include load times is around 100ms for the full range of seeds, or about 6 seconds for all 21 sockets and 4 jewel types (84 total searches))

patent notch
hazy fiber
#

5-15 seconds per jewel socket or total?
on phone atm (should be sleeping) so cant look properly

#

a tradeoff for having nice csv/xslx data files people can just look through

patent notch
#

total

hazy fiber
#

thats not too bad

patent notch
#

let me show you here one sec

hazy fiber
#

its useable enough, and better interface than mine (mines meant for loading weighted sums into it atm, and needs a UI hence just released datafiles and not searcher)

hopfully helps a bunch of people

grave wren
#

the simpler jewels take a few secs for me too including generating a trade link but its cli only so far

#

eternal jewels take ages

worthy cape
#

An... eternity? πŸ˜›

quick dragon
#

with the datafiles, I can search through 21 notables and filter +sort them based on weight in 100 ms

#

also, heard there were issues with militant faith and elegant hubris that I'm trying to track down? if anyone has info let me know

one thing to note with the https://github.com/Regisle/TimelessJewelData github is that it states replacements are index_of_Change = _rid - 94 in alternate_passive_additions.json rn, but should be replacements are index_of_Change = _rid + 94 in alternate_passive_skills.json which will be updated once regis back on pc

#

ive got my searcher working, so trying to figure out where the disconnect might be if there is one

eager rivet
#

Hey, I created an API for timeless jewels using the data dumps of KeshHere. Feel free to use it:
https://jewel_api.folx.workers.dev/BrutalRestraint/1555

tepid pawn
#

So I've gotten my own search tool mostly working, but now I want to check radius and stuff. Is there any data with just xy coords or would that be too easy XD?

trim lotus
#

@tepid pawn If you name a Format, I can convert the poedb info on passives in radius for you

quick dragon
#

wait nvm that doesnt have it either. bad memory

#

sorry

tepid pawn
#

That was what I was working with yeah

#

I can work with basically any format that makes some level of sense.

quick dragon
#

png please 🀨

worthy cape
#

Sequential outputs from PRNG πŸ˜‰

grave wren
#

pico8 gamecart

trim lotus
tepid pawn
#

Do all of those but record the binary in morse code and send that through a voice chat

tepid pawn
quick dragon
#

make each value a coefficient in a polynomial and send me the roots

#

no

#

play the function as a sound

tepid pawn
#

At that point just encrypt it with rsa and forget to give me the public key

quick dragon
late kestrel
worthy cape
#

Well, it's still inside together with the game source and assets, because I still don't know my passphrase πŸ˜„

tepid pawn
#

Sounds like one of those stories that only becomes funny after years have passed

late kestrel
#

On the scale of "never funny, funny once, always funny", it rapidly shifts between the leftmost and rightmost ends of the scale.

worthy cape
#

Ended up reverse-engineering the baked assets I still had lying around and essentially doing a new implementation from scratch.

#

Kind of funny, a lot of the work I did back then came in handy when figuring out PoE's animation file formats now, got some flashbacks ^_^

quick dragon
#

yep. the data is fine, just the name is wrong. if you rename them then they should work

#

sorry bout that. got baited by a comment with the wrong indices

trim lotus
quick dragon
slim pilot
polar wren
trim lotus
#

@quick dragon are you really writing or is my discord bugging out?

quick dragon
#

fuck me it didnt attatch the zip. one sec

#

https://pastebin.com/LhpcE7XQ is a simple weighted search function in c# you can slot into whatever application you've got already (with hopefully only minor edits) that will work with the new data files. It does not translate between node names/ids and indices (those can be found in the node_indices file) and it does not translate between stat index and stat name/rid (those can be found by following definitions in the github below). also note that the sort is currently inverted to make it easier to read the best jewels from a console app, but you might want it in a "normal" orientation. just change searchResult.Sort((y, x) to searchResult.Sort((x, y)

https://github.com/Regisle/TimelessJewelData has definitions for everything, though wile regi is offline, itll be slightly out of date. importantly,
replacements are index_of_Change = _rid - 94 in alternate_passive_additions.json
should be
replacements are index_of_Change = _rid + 94 in alternate_passive_skills.json

with the datafiles and this search function (or one similar to it), you can get the time to parse, filter, and sort for every seed and every notable within range of a jewel socket down to around 100ms

we're working on getting glorious vanity into a nice format too, but as I'm sure many of you are aware, that jewel sucks and its results cant be as cleanly compressed and seeked through. we think we have a decent format that will balance speed and file size though, so be on the lookout for that

πŸ“Ž ProperlyNamedDataFiles.zip
slim pilot
#

damn C is really nice for shit. gz.

fallow zinc
#

@patent notch what is that written in?

patent notch
#

AutoIt3, its like AHK but i know it more than ahk so easy to use for me.

fallow zinc
#

the clunky af script that I made has pretty much the same features as yours but it's more manual, it runs through all seeds in a split second

#

oh ok

trim lotus
orchid light
#

Does anyone have the change id from start of sentinel league?

frank drift
quick dragon
#

glorious vanity data file is (basically) done. came out to ~43MB, 20 when compressed. brutal 4x the size of the other files combined, but still manageable

vapid pulsar
#

Hey does anyone know much about the skill tree Json?
Seems like the group Y coordinates need to be flipped or the tree renders the groups incorrectly

neon plume
#

Considering that it's WASM-based, feel free to embed it into some trade extension

frank drift
#

grumpy Novynn

quick dragon
#

demospacebonk stop querying that demospacebonk

jagged dew
vapid pulsar
#

I have (0,0) as the center of the tree.

for example group 93 has (x: -6150.77, y: -6644.11) but appears at the top left of the tree

#

and group 231 is down the bottom and is (x: -2783.94, y: 8089.25)

#

If I multiply y by -1 my tree renders fine but it seems like a werid thing to do

jagged dew
#

Group 93 contains node Blunt Trauma which should be at top left of the tree though, no?

vapid pulsar
#

Yes that correct. I would assume that a positive y is up, not down?

jagged dew
#

Ah no that would be math I think. for some reason we mean down most of the time when programming

vapid pulsar
#

I think is why. Im using a Cartesian coordinate system

unknown.png
jagged dew
#

Yep so you'll have to flip before rendering (or maybe you could when reading the json I guess)

vapid pulsar
#

Yeah, cool. Im not going insane then droxthumbsup

hazy fiber
#

k fixed issues with wrong file names

orchid light
#

Is there anyway for me to find harvest craft weighting?

hazy fiber
#

@fallow zinc sorry the GV is zipped, so if you want to use it, youll have to find some way to unzip it

vapid pulsar
orchid light
vapid pulsar
#

Oh do you mean the application of craft? Not the spawn chance of a crafting option?

orchid light
#

Yes that

#

Its the only data thats not shown on craftofexile in the main tab ?

vapid pulsar
#

There is a weight per mod. It's held within the Mods.dat

orchid light
#

So those mod weightings are irrelevant of the crafting method used?

#

And only being able to be removed from the craft by blocking?

vapid pulsar
orchid light
#

What i mean is, for chaos spamming it uses those weights, and for harvest crafting it also uses that weights but the behaviour changes based on the type of harvest craft for instance, reforge speed, rolls a speed mod, and then the rest is a chaos spam?

vapid pulsar
#

Yes for a chaos spam it just uses the weights as they are.
Harvest, depending on the craft used will filter the pool and or alter the weights. So something like reforge more common will 10x the weights of the mods of that tag

#

If you want more in depth info about specifics it would be better to ask the crafting experts. This stuff is more game mechanics that tool dev questions.

orchid light
#

I was really under the impression that the weight was different for harvest but i guess that does not make sense

#

Ill ask around the crafters aswell

vapid pulsar
orchid light
#

i guess i overcomplicated it very much XD

orchid light
#

Im still trying to figure out how affix counts are choosen?

#

@vapid pulsar is there a specific channel i should ask for talking to crafters?

#

Also is there a better place to get the affix data besides RePoe?

frank drift
#

RePoE is just repackaging of data from the game files

#

you can read the game files directly if you really want to

#

there's a reason harvest isn't on the Craftofexile calculator, it's more complex to calculate so it's only in the Simulator

orchid light
#

Yeah my problem is RePoe is packaged only for python and im looking to do this in Scala

orchid light
vapid pulsar
orchid light
#

I asked in tft discord

#

Yeah json is also going to be a pain since Scala is strongly typed.

#

I might just write a generator for Scala like PyPoe

vapid pulsar
#

Json is a pretty common format i would image that most languages have some library available to load it in? play-json comes up with a quick google for scala

frank drift
#

you should be able to use the existing json yeah, unless you need something that hasn't been added to RePoE yet

orchid light
vapid pulsar
#

Sounds simpler to me than writing a generator. But eh, that sort of stuff can be fun to do.

orchid light
#

Well once thats done i can use it, being a programmer is always about spending weeks optimising 5 hours of work

#

The reason i need it in Scala is because Akka Actors

velvet fog
#

https://www.pathofexile.com/character-window/get-account-name-by-character?character= seems removed

hazy fiber
#

yup, someone mentioned it before

hazy fiber
#

with the new patch

velvet fog
#

Ouch, too bad.

frank drift
fallow zinc
#

thanks @hazy fiber - zip is totally fine

#

I'm refactoring first pass of the code now that I understand how it all works

#

idk exactly how much more effort im going to put in though as it has been done by others already

#

might just keep it private

potent skiff
#

hey all, new Poe-Dev here i guess. as a hobby i thought i try extracting the GGPK info like pypoe does, but in Rust (new language to learn)
now i got as far as extracting all files and putting them in the related PDIRs, but with the bundles i am not fully sure.
https://zao.github.io/poe-doc/bundles.html talks about an index.txt, which i cannot find, only the index.bin. any pointers?

worthy cape
#

The text file has changed name lately and is not needed, you can reconstruct the full file listing by processing the directory representation at the end of the index file.

potent skiff
#

aah look at that, thats something new. thanks a lot this helps πŸ™‚

formal kraken
potent skiff
#

not yet. it's very messy code, learning rust at the same time. a lot of copys and borrows etc done wrong i guess ^^

#

but looking at @worthy cape s link i see that it's now only in the index.bin, that i have, with that i can work πŸ™‚

worthy cape
#

On Steam the old index txt is now Bundles2/_.files.log.txt.

potent skiff
#

yep that i have, even on non steam client. but seems i dont need it (and it does not have the info which bundle has which files as it had before apparently) https://imgur.com/9U9ZV5x

#

as per the link you sent me

#

so thanks so much. this helps a lot

worthy cape
#

In other news, I can't believe that my data service is finally fully automatic from releases published on Steam to loose files directly available over my network.

potent skiff
#

did you ever extract all of it? just doing the files so far it takes me nearly 1 hour ^^

#

and gz

worthy cape
#

All the files, all releases since 1.0.0.

potent skiff
#

well the whole ggpk as it is (standalone from me). until step where you do the bundles i guess because that i have not done yet ^^

worthy cape
#

Last I counted there was around 250 million files with 1.93 million unique file contents.

potent skiff
#

so no ooz yet

worthy cape
#

A release is at around 750k bundled files so if you do it suboptimally, a hour is definitely in the ballpark.

potent skiff
#

yep, makes sense, i have no idea to parralellize yet as a file read is pretty much not easy there

worthy cape
#

For GGPK the trick tends to be to do a linear scan first to collect all the chunk metadata and then use that when reassembling the directory structure.

potent skiff
#

and as i do random reads with the offsets.. well no buffering. and i cant load 32 GFbn into ram

worthy cape
#

Like you read a chunk header, store that metadata and skip ahead by the chunk size to read the next chunk header.

potent skiff
#

yeah, might do that later

formal kraken
#

mmap2 crate iirc

potent skiff
#

okay worth a look thanks

formal kraken
#

*memmap2 actually

potent skiff
#

but first i will try getting it all out for now, so bundles to files, and somehow i guess implement ooz compression

#

because i dont want the external dll, nothing unsafe if possible ^^

worthy cape
#

We all want pure and more understandable ooz code, but that's a big project.

#

I spent a good few days the other week debugging some compressed data that proper Oodle could decompress but ooz couldn't, had to step quite deep into it to find out where it broke and to fix it.

potent skiff
#

i know. but as it's a hobby i will try and make some stuff. and give up if it's not in my range ^^

worthy cape
#

You're talking to someone who thought that a free-standing hideout editor would be doable πŸ˜„
I'm years into that project now.

potent skiff
#

haha

worthy cape
#

The best thing about naively getting started is that you find all the fun stuff out along the way.

potent skiff
#

i mean for me als old java dev with pretty much no low level experience i got quite far already in a new language, i am happy so far πŸ™‚

worthy cape
#

Probably on the third rewrite or so of my PoE stuff, going from C++ to C to Rust to C++ with a bit of Python inbetween for some tooling.

potent skiff
#

why isnt your exporter easy to find if it works? i wanted to try some stuff, pypoe didnt want to, so i said, "let's do my own". i think some poeple would be happy to use it no?

worthy cape
#

bun_extract_file is the tool that I recommend for people who wish to extract files for further use.

#

The Bun part of that repo is C designed to be used (poorly) via FFI for use in tooling.

potent skiff
#

oh okay

worthy cape
#

Most recently I've pivoted my tools (and did some proof-of-concept in PyPoE) of remote file access from my own data servers.

#

This primarily because I wanted to have specific revisions at my fingertips when developing rather than juggle GGPKs around, as well as comparing data across releases.

potent skiff
#

elaborate? updateing data directly from a new ggpk?

#

ah

worthy cape
#

I unpack each release, hashing all the files and storing out an index mapping file entry to content hash.
With that index for a release a client can obtain and cache individual files it needs.

potent skiff
#

i want to go the ease of use route (thus the grand plan on implementing ooz myself) for a single app to extract at least all files, later i could go the json output route from there

#

but again, maybe i just give up. it's more to learn some rust

worthy cape
#

Turns out that downloading hundreds of thousands of individual files over HTTP isn't a good and scalable idea ^_^

potent skiff
#

haha

worthy cape
#

Pivoting now just the last few days to downloading individual bundle.bin files to my clients instead, having them do the decompression and slicing again.

potent skiff
#

maybe easier with websockets, but then again, i am no web dev

worthy cape
#

Yeah, WS would help with pipelining and handshaking.

#

I've mostly disregarded such approaches as you get a lot of HTTP caching for free with proxies like Cloudflare.

#

(found out recently that CF ToS doesn't quite allow non-website content so that idea fell out too πŸ˜„ )

#

Much like you, a lot of the PoE-adjacent dev I do is to learn stuff and occasionally something usable by the community falls out.

#

I'm currently thinking over whether it'll be a Good Idea to gut my VFS code and replace it with a dedicated I/O and decompression thread, rolling my own futures.

vapid pulsar
#

@worthy cape What where your reasons from moving away from Rust back to C++?

worthy cape
#

Crustacean allergy πŸ˜‰

#

I've got a list, just gotta clean out lunch.

#

There's a few reasons, some about the language + ecosystem and some personal.
The ecosystem is a bit immature and full of friction, especially when it comes to bindings for native libraries. Many bindings change their interface every few releases in an attempt to find an ergonomic API that works with the borrow checker while still trying to let you use the library as originally intended with as much of the underlying capabilities as possible.
The graphics and rendering ecosystem is inbred and extremely focused on Vulkan, leaving you shunned and reinventing a lot of stuff to use anything else than the Blessed Gfx Way with the IRC dictators.
Bindings also quite often lag behind upstream, making it a whole ordeal if you end up needing something like the tiling branch or patches to Dear ImGui or a more recent base for a -sys crate.

#

Personally, I got really fed up by knowing what kind of code I wanted to write, only to be constantly hamstrung by how it just couldn't fit into the lifetime structure of idiomatic Rust, either due to library vendors making choices for me (sqlite3 transaction lifetimes making them near-useless around loops) or because lifetime is more complicated than the ways allowed by the language and trying to shoehorn it into it was way too much focus on noodling workarounds than actually writing and exploring code.

#

Also personally, C++ is home, in the end I wish to be more proficient there than in a language that just isn't fit for my purposes and where everything seems to be in flux, especially if you look at things like the async storyline.

#

FFI is more cumbersome than it ought to be, libraries like nom fundamentally changing every time you look at them and all documentation being extremely stale, stray outside the scope of Blessed True Libraries for things like serde and it's a wasteland.

#

All in all, neat language, not for me, I wish anyone trying to use it for Actual Things all the best.

vapid pulsar
#

Yeah I can definitely see how fast a lot of crates change and break. The bindings really are a nightmare.
I guess I don't use the language much so haven't got super fed up etc with the smaller scale projects I've done with a lot of those issues you have outlined.
Interested to see where the language ends up in a few years.

#

People at my work are currently pushing for golang. that will be interesting.

worthy cape
#

I hope the peeps at your work don't ruin everything πŸ˜‰

#

Go is definitely one of the language I just won't touch based on my interactions with people on the bug tracker and the expletive in charge making the community extremely toxic.

#

Doesn't help that it's extremely Unix oriented and involves a lot of manual work to do anything in code, but that's old stuff.

#

For the peeps who tolerate Golang and find it useful, yay for them.

vapid pulsar
#

We will probably stick with TS for the foreseeable future. But yeah not the keenest on it.

And yeah each to their own, if someone like writing in X random language, doesn't bother me either.

#

I was curious as usually you hear of migrations to Rust from C++ not the other way around.

worthy cape
#

I've been going back and forth a few times by now and it almost always ends in sadness over the state of things.

polar wren
worthy cape
#

I had a legitimate problem in that the Go stdlib's TLS implementation rejected handshakes with somewhat large but still within RFC lists of acceptable CAs.

#

This arose in a real-world system where I was looking at implementing part of it in Go, but the response on the tracker was pretty much along the lines of "something like that cannot exist out there, change your world".

polar wren
#

oddly familiar templarLul

worthy cape
#

The code rejected these handshakes as some sort of shitty DoS resilience as "clearly no-one will ever have handshakes larger than 2k".

polar wren
#

let me guess, the official docs were equally unhelpful

worthy cape
#

Mine were like 8k, and the RFC itself suggested at least 12k.

#

As for "changing your shit", good luck doing that when you're not Google.

#

You can't exactly change the fundamentals of the whole gosh-darn IGTF, the whole point of the federation is distributed trust across many CAs.

formal kraken
#

I think I've seen that templarLul

worthy cape
#

I'm told that it's less bad now that Filippo is on that team, but it was a dealbreaker event for me to ever consider the language again, and every time I hear about the language it's some smug bastard telling people to get good.

buoyant dagger
#

Btw re: the recent changes to the character and account api calls, i just got a non answer on my thread asking about whether it was intended or an oversight, so i guess they're here to stay

simple ravine
#

I might learn a new language just for the heck of it. Thinking about trying Rust

slim pilot
#

Learn Go, it sucks, it is boring as fuck, but when you finish a code it really gives a huge satisfaction

#

knowing that you will never ever code in Go, at least for 2 hours

slim pilot
worthy cape
slim pilot
quick dragon
#

fr though rust is a solid choice. go for it πŸ‘

potent skiff
#

sadly that is still true, got people still learning"mainframe" at the bank i am working

quick dragon
#

yeah, all those old systems from however many years ago now that just cant afford to risk updating due to their criticality...

frank drift
#

*due to not wanting to spend money

neon plume
#

so, I managed to draw the tree and calculate the paths and jewel sizes, but I have an issue, the jewels include/exclude ones that it shouldn'd/should. am I missing something or is math just wrong?

unknown.png
#

the easiest place to see this is on the top right of the tree, where bottom node should be excluded, but right one should be included, but the bottom one is... closer to the jewel socket by center distance

unknown.png
#

if I increase the size, I include the bottom, but exclude the right, which is wrong

timber path
#

How does one determine if a specific mod is an incursion or bestiary (aspects etc) mod? I can't seem to link any column in the Mods.dat to something related to Incursion

late kestrel
carmine merlin
timber path
carmine merlin
#

Could ask the dev of CraftOfExile

#

@nova hamlet ^

neon plume
late kestrel
#

It's a bit of a pain. I copied some of the code from the pob github. You can try overlaying your output with official ggg output to make sure you're getting it right.

nova hamlet
#

@timber pathThere is nothing specific in the dataset that can allow the detection of incursion group mods other than the fact that their name field have something incursion related, like "of Citaqualotl" afaik.

neon plume
#

I'm not familiar with PoB source, do you know where the tree rendering occurs?

carmine merlin
#

PassiveTree.lua is where the tree stuff is done

neon plume
#

Yep, found it, thanks. Still not sure how where I've messed up as even the center-most group is wrong for me

unknown.png
unreal solar
mental edge
#

is there any guide / documentation about using poe trade api ?

tepid pawn
#

Welp. There goes my chances of actually finishing my tool. I decided that I should have extremely good version interop and now my code is this...

unknown-41.png
#

I love C# but people keep telling me I am using the language wrong when I set a "No classes" rule. Idk why...

neon plume
#

ok, at this point I am assuming I have gone fully insane, nothing makes sense, even debug drawing over the image doesn't help. The group 329 at orbit 3 (which according to skillsPerOrbit can contain 16 nodes) has 6 nodes placed at 1, 4, 7, 9, 12 and 15. But they don't match the locations of the official skill tree, but you can see they are perfectly equally distributed...

unknown.png
#

and I just realized where I went wrong, thanks for being my rubber duckies

#

...and it's perfect

unknown.png
velvet fog
neon plume
#

then I realized how dumb that was

#

should probably sleep

velvet fog
#

orbit angles have special values on 16 and 40

velvet fog
frank drift
#

that the one from reddit?

velvet fog
hazy fiber
#

not showing roll range on Glorious Vanity? (or what Might of the Vaal/Legacy of the Vaal give?)

velvet fog
#

that's on my to-do list

slim pilot
#

dang shit timeless jewels will be expensive next league

potent skiff
#

haha great news for me, i liked sentinel->legion farming and selling emblems. price might rise then πŸ™‚

slim pilot
#

i will try to replicate passive skill tree display of poedb and delibaretely fail today lets go

silk pollen
slim pilot
#

Can anyone help me to generate a passive skill tree from JSON? I tried to load every data to structs (https://pastebinp.com/5yO2rU8DDaLdrLeomdxJmg#Dx9NJi4O8WsM0XKsX3IVqE_XhMxL0u2caVizbwRJy3g=) and work from there but as every subgroup of the datas have different names but same variables it just didnt work and I dont want to work with databases. Any short way to generate, or create it? (https://github.com/poe-tool-dev/passive-skill-tree-json/blob/master/3.17.0/SkillTree.json this data right here)

velvet fog
slim pilot
#

just a simple thing to represent the nodes

#

built struct with someones help, gotta figure out the rest. even pseudocode would be fine at this point

velvet fog
#

could it draw with svg?

slim pilot
velvet fog
slim pilot
# velvet fog

yeah got what you said, thanks for svg tip. will work on this a lot today, thanks again ss is just a trash code for test

Ekran_Resmi_2022-07-05_13.58.42.png
velvet fog
slim pilot
#

shit is... confusing.

slim pilot
#

yea fuck this

#

this tree shit is literally madman job

#

fucking salute to that pob guy

patent notch
hazy fiber
#

How did you get GV to be only 6MB? smaller than what lethal etc are?

#

missing the small nodes, not recording the value as well as the change, and only 1 change not the up to 4 it can cause?

#

or just happen to have good compression ratio on it?

oak estuary
mortal bone
formal kraken
mortal bone
#

there is no api for the skill tree json

formal kraken
#

no but the models are the same

#

pretty sure I copied a bunch of fields from there into my models

mortal bone
#

I don't see the models, but I could be wrong

worthy cape
#

Hrm, that's the user's passive tree.

mortal bone
#

that is the encoded data, yeah

worthy cape
#

Ah right, PassiveNode just above.

mortal bone
#

Oh, right about haha

worthy cape
#

😎

formal kraken
#

yeah was trying to link that, but no id on PassiveNode

#

not a lot though

mortal bone
#

That is also only 2 of a bunch of objects πŸ˜„

formal kraken
#

yeah thought it was more 😦

arctic knot
formal kraken
sharp sinew
#

hey im having issues with poe awakened

#

using the price check hotkey doesn't display anything until i alt + tab where i get a glimpse of the menu before disappearing

#

im running the game in windowed fullscreen, i've tried fullscreen also. I play at 1440p if that makes a difference

silk pollen
#

If anyone is interested in a C# interactive application to mess around with timeless jewel data in C# using LINQ to SQL, please let me know and I'll look into setting up a proper example/instructions and publish it somewhere.
Only for developers, you need understanding of C# and LINQ to work with it, but if you do have the knowledge you can use the power of LINQ to query exactly what you want without UI limitations.
Haven't bothered with Glorious Vanity data yet, so that's not supported.
I'll include my timeless brute force application that will fill the database (seed/jewel sockets/notables) for all non-Glorious Vanity seeds within about 5 minutes (admittedly on a solid machine, amd 5900x) so shouldn't need to provide any data.
Feel free to DM or tag for questions.

pallid doveBOT
#

Please don't post discord invites

tepid pawn
#

Oh no. you provided speed information. Now I feel competitive for no reason whatsoever.

silk pollen
#

it's just to state that there's no need to share all the data, chill

tepid pawn
#

Hey I gotta have fun somehow ;)

mental edge
#

can anybody help me how to make 2d array for Data for glorious vanity ?
i don't really understand how to start this in python

patent notch
hazy fiber
mental edge
hazy fiber
# 
local GV_nodecount = 1678
local seedSize = data.ConqSeedMax[1] - data.ConqSeedMin[1] + 1
data.timelessJewelLUTs[jewelType].sizes = file:read(GV_nodecount * seedSize)
for i = 1, (GV_nodecount * seedSize) do
    data.timelessJewelLUTs[jewelType].data[i] = file:read(data.timelessJewelLUTs[jewelType].sizes:byte(i))
end
tepid pawn
#

so idk what others have done for glorious vanity, but basically I found that the best way to store smalls and notables is just to treat each roll as its own node

#

The ranges for the smalls are not massive, and by my math there are 171 possibilities including rolls

hazy fiber
tepid pawn
#

except the notables if you include the weird ones right

#

Including might/other one of the vaal I came out with 30-31 bits

hazy fiber
#

might/legacy of vaal? those are size 6/8 (3 or 4 stats, each with a roll)

#

you can compress them further, (zip size is 20MB vs 43 unzipped) but kept all sizes at full byte for speed

tepid pawn
#

Yeah thats what I am doing as well

#

What ordering have you been using in the binary for nodes?

hazy fiber
tepid pawn
#

I was thinking about doing a sort to try and keep things continuous for each socket but overlaps make that pretty complicated

hazy fiber
#

only GV uses the small nodes, the rest just use notables, sorted by graphID with the notables at the top

#

um, nodeID -> jewel seed, so that SEEK grabs the largest block of useful data to speed up weighted searches

tepid pawn
#

Next thing on my todo list is a heuristic for jewel quality

#

So for example, there is a seed for militant faith that gives 6x thoughts and prayers which is statistically a pretty big outlier in terms of repeated notables. (seed is 54xx off top of head, location is the socket near shadow)

That seed may not actually be good because the position is so far from templar start, but its still interesting and it would be cool to have a list of the interesting seeds similar to that one.

hazy fiber
tepid pawn
#

You can easily hit 150 because of how dense nodes are there

hazy fiber
#

but yes interesting outlier

tepid pawn
#

I checked XD

#

You hit 150 just traveling to the notables you want

neon plume
patent notch
neon plume
#

Thanks, took a while to get the tree right, and found a couple places where GGG cuts corners (sometimes literally)

#

...now to implement reverse searching

violet path
#

I think @patent notch just got that in theirs sometime today

trim lotus
worthy cape
#

@neon plume When scrolling in/out, when reaching the innermost zoom level the tree jerks violently to the northwest instead of zooming around the cursor.

neon plume
#

Yeah, I know, zooming is real bad

worthy cape
#

(Firefox, Windows)

#

^_^

neon plume
#

I couldn't figure out the math on that one

#

too many offsets from too many places

patent notch
#

For me terms reverse searching and searching are opposite i guess.
In my mind searching is for searching the jewel you want and
reverse searching is checking the mods that jewel you have has on different nodes.
This might get confusing later lol
@violet path ya just released mine 3 hours ago on github

#

I am not a web dev so I stuck to what I know the most about and made local program πŸ˜„

neon plume
#

The actual calculation is done in Go, I complied to WASM to make it accessible to anyone on any platform

patent notch
#

No idea what WASM is tbh πŸ˜„

trim lotus
#

WASM = webassembly

neon plume
#

allows you to compile languages like Go, C++, Rust, etc to something you can ship and execute in the browser

patent notch
#

I see, I am programmer by hobby not by degree so some terms fly over my mind and have to google it lol

neon plume
#

haha, I dropped out, and even then we weren't taught anything like that

trim lotus
#

everyone that is any good(not talking about skilled) is programmer by hobby even with a degree

patent notch
#

Googling is a skill more people need to develop than anything to become a good programmer/coder lol

trim lotus
#

A colleague of mine is a bachelor student and yes, he lacks googling skills πŸ˜„

patent notch
#

I have googled simple for loops because I got confused between JAVA, C#, Python, Autoit and it all got jumbled up lmao

trim lotus
#

jumping languages is pure terror, java has -> for lambdas while js has =>

patent notch
#

one needs : and one needs ; and what not lol

#

I'd rather google it in 2 second than stress my brain for 30 seconds lol

neon plume
#

and then one requires you to indent correctly otherwise it will blow your brains out

patent notch
#

fking python i knew it

neon plume
patent notch
#

so angry at indents, too much anger inside sorry lol

#

like 4 spaces and 1 tab gets mixed up when copy pasting like wtf

worthy cape
#

No offense, but may I indent your code? πŸ˜›

trim lotus
trim lotus
neon plume
neon plume
#

I should probably fix the linter rather than rushing the tree out the door

worthy cape
#

(just a great joke/insult to use on others)

patent notch
#

hehe

neon plume
#

there, all fixed πŸ˜„

worthy cape
#

Has anyone done any toying around with efficient bundled file access?
I've moved my code recently from using loose files to using bundles downloaded on demand but processing throughput is quite meh as you need to decompress whole 256k chunks even for small files.
In the bulk extractor I could just order everything by bundle up-front to get some reuse from chunks, but in a real program I'm having some trouble batching up enough file reads to do so.
I've been thinking about keeping a cache of recently decompressed blocks but I have no good eviction criteria for it and the game data is way too large (50 GiB) to cache most of it.
Amusingly enough performance gets worse if I add more I/O threads as requests for blocks get even thinner.

neon plume
#

If you used some sort of indexed compression method, you would only need to read the header and then jump to the indexed position in the bundle, decompress N bytes, and get your asset

worthy cape
#

The compression scheme is locked-in as it's the Oodle that PoE's datafiles comes in.

#

Each bundle has the uncompressed payload split into 256k sections each individually compressed and indexed.

neon plume
#

any possibility of building an index outside of these files and referencing that?

worthy cape
#

I've got indexes with the exact offset+size of all files in each bundle, the problem is that you cannot decompress anything smaller than a full block.

neon plume
#

I have no idea how PoE files are compressed, take my suggestions with an ocean of salt

#

ah, ok

worthy cape
#

My application code works by file path (or hash of file path), asking my VFS to open and read individual files. The VFS consults the index to find the bundle and blocks it needs and slices out the resulting data for the caller.

#

Right now that's extremely synchronous and the only real way I could see this scaling up much is if I somehow open more files asynchronously in application code and process them as they complete.

neon plume
#

what tool is this for?

worthy cape
#

A bit of a problem there is that loads are typically quite serial in nature.
If you're loading say a skinned mesh, you first load the .aoc for the object definition, which gets you a .sm for the skinned mesh and an .ast for the skeleton, and loading those gets you a .smd for the actual geometry, etc. That's fairly linear.

worthy cape
# neon plume what tool is this for?

Right now it's mostly for my "hideout editor" and "asset viewer", but also for my bulk parsing tools to test my format loaders across all files.

#

Two quite disjoint use cases where-as one loads a chain of files in order and determined by previous load results, while the other shotguns the whole world.

neon plume
#

if you can, try preloading any assets you expect you might need async, and then join the loading thread when you actually need them. I wrote a tool that parses binary UE4 pak files, and I noticed that fetching each asset individually was actually slower than requesting bigger blocks of data and using them if needed, dropping from memory if unused

worthy cape
#

Yeah, preloading and batching are probably key, just confused as to what kind of interface I must expose to my application code from the generic library and how on earth to reshape the very linear loopy code into something that's more deferred.

#

The bundles suspiciously named _Preload are probably a good candidate as it only expands to just above 100 megs.
_Tiny ends up almost a gigabyte.

orchid light
#

On the RePoe data i see a field called domain . Are all the domains stored somewhere or should it be inferred from this field?

oak estuary
orchid light
oak estuary
#

Correct. I believe there used to be, but it was pruned from the client side data (I could be wrong about that)

orchid light
#

and all domains are linked together throughout the data. There isn't a domain field linked to something else?

oak estuary
#

We are talking about mod domains, right? The column in Mods.dat? It's just on that table, iirc

orchid light
#

base_items.json also has it?

oak estuary
#

Yeah I guess it does

orchid light
#

Is it the same domain field?

oak estuary
orchid light
oak estuary
#

That's a typo...it's also in a comment. It's four years old, too

velvet fog
#

ModDomains only have header and removes everything to prevent data mining

orchid light
oak estuary
#

Should be

orchid light
#

progress has been made, that modifier number is lower than i expected

unknown.png
patent notch
#

@ebon oasis got it

ebon oasis
#

\o/

worthy cape
# neon plume if you can, try preloading any assets you expect you might need async, and then ...

Did some heavy lifting with off-thread bulk load requests sorted by bundle, makes the calling code a bit more convoluted as it has to batch up all the file opens but it runs adequately, the below being from opening all the AOC files, followed by opening all the SM files referenced from those:

2022-07-07 15:18:35.865 (   3.120s) INFO| Bulk load of 44790 files begins on worker 0.
2022-07-07 15:18:36.237 (   3.492s) INFO| Bulk load of 44790 files ends on worker 0.
2022-07-07 15:18:36.812 (   4.067s) INFO| Bulk load of 26629 files begins on worker 0.
2022-07-07 15:18:37.048 (   4.303s) INFO| Bulk load of 26629 files ends on worker 0.
neon plume
worthy cape
#

Compared to decompressing blocks over and over again for each file? Magnitudes.
Compared to before where I had pre-sliced files on-disk that just needed to be read and decompressed with zstandard, probably around the same ballpark.

neon plume
#

That sounds pretty good then

worthy cape
#

Big pain points now is enumerating all files in the directory tree, as well as the legacy code that accesses larger numbers of files individually.

neon plume
#

Also, in theory, you could try making some priority queue with N readers, and if queue is empty, you just feed it previously-unparsed blocks

worthy cape
#

The manifest I have for fast file lookup only has path hashes so for enumeration where the caller wants paths or at the least file extensions, it has to look up paths from the database for all 750k entries.

neon plume
#

damn

worthy cape
#

Some of these lookups are a bit unnatural compared to a regular application, like "give me all the AOC files" or "give me all the DGR files" or in the case of the bulk parsing tool, "give me all the files".

#

Some lookups are reasonable tho, as the material compiler needs "all FFX files", but that might make more sense as a direct generation of paths from scratch from the directory tree metadata.

#

The way that the bundle index in PoE represents file paths is in a compact encoding where it concatenates common substrings together to form full paths for a specific directory (and optionally its subdirectories).

#

Ho ho, this is rewarding.
On a clean machine on my LAN it takes around 16-20s to start the asset viewer as it downloads and processes bundles from scratch from the data server.

#

Next boot sub-second to get to interactive UI and ~4s to load all object metadata.

neon plume
#

not bad

neon plume
frank drift
#

I've been using it to check seeds from other tools, it's nice

#

the reverse search is a little too limited to be useful to me compared to other tools though

grave wren
#

Is there a good weighted search tool now for a specific jewel socket?

frank drift
#

at least one private one, dunno about public

grave wren
#

Might just have to add weighted search to my tool then but I guess the jewel I want just doesn't exist anyway in league

neon plume
frank drift
# neon plume oh, what do you feel is missing?

the ability to search by transformed notable name rather than just effect, the ability to specify required number of notables (eg >= 1 of X, >= 1 of Y, and >= 2 of Z), the ability to limit to specific notables rather than all in range (many cost too many points to be feasible) or limit to a maximum amount of passive points, the ability to use weights

neon plume
#

oh, is there a tool that has all this?

frank drift
#

"timeless jewel finder" does half of it and another does the other half

neon plume
#

I mean, most of that seems pretty easy, the hardest would be the point limits

frank drift
#

point limits would require input of existing pathing yeah

carmine merlin
#

@frank drift PoB should do all of that once we’re done. Not sure when the update for it will be out though

hazy fiber
#

uh "per point" will probs not be in the initial implementation, but maybe

late kestrel
#

It's not a critical feature, but impossible escape options are also very nice. People like their weird thread of hopes.

silk pollen
late kestrel
#

I meant it for someone trying to make an all-in-one tool. I have my own local tool that satisfies all my personal needs.

neon plume
#

hmm, considering I went through the trouble of drawing the tree in the first place, might try and tackle that next

pliant cipher
#

Hey yall, is there a list somewhere of the various trade api endpoints for static data that the trade site uses? I've found four, api/trade/data/leagues,static,stats, and items, not sure if there's other useful ones out there before I go building my tool

worthy cape
#

I/O systems are fun.
Added a different type of load to my VFS where rather than one-off loads off-thread or bulk loads off-thread, there's also the ability to issue multiple loads interactively on-thread bringing your own block cache.
That way you can have loopy and unbatchable logic that is likely to touch related files (like loading levels) do reasonably compact and efficient loads without burning blocks at the expense of having to throw away the cache once you're done.

#

Golfed the function that loads the tileset and mesh data for a level graph from 1s-1.9s to 60ms.

slim pilot
#

endpoints are still working but testing is dead i assume.

pliant cipher
#

Oh, perfect.

#

thanks.

slim pilot
#

np

worthy cape
#

Mandatory note that the trade API isn't an officially supported API and may change without notice.

pliant cipher
#

that's fine

#

I'd expect that

worthy cape
#

πŸ˜„

slim pilot
#

i was writing my old poe code in python from fresh in go because it was spagetti

#

now this new code is much more fucking spagetti someone fuck me pls

pliant cipher
#

Also, just curious if anyone knows GGG's stance on in-process overlays? Like discord's in game overlay or overwolf that hooks directx draws to render their UI to the window. Figured if I'm going to make a tool similar to awakened trade it might not be a bad idea to just skip the windows positioning funk and just have it draw some of the elements with ImGui or similar on the canvas. Haven't seen other tools do this so I'm guessing its probably not allowed

worthy cape
#

We strongly discourage them here, not sure if there's ever been a public stance on it but it's way way riskier than the typical window-on-top APT style overlay.

pliant cipher
#

yeah

hazy fiber
#

Dont know if theres an official stance, but if no one has done it then probs a good idea to ask directly or stay away, as zao said, way riskier

pliant cipher
#

I haven't even started building anything yet so just curious, if I decide to investigate going that route I'll ask someone

#

that stuff is always a... very grey area lol

worthy cape
#

Executable apps that interact with the game or game files.
This behaviour is strictly against our Terms of Use (section 7b).
Creation or use of these type of applications will result in immediate account termination.

#

While APT and other action-based ones fall under the second class of independent apps.

pliant cipher
#

Still probably a question for someone from ggg because discord HAS to interact with the client to draw it's overlay in-game, so am I liable for a ban if I turn it on? (I'm not arguing with you, I agree, just curious if this has come up before)

neon plume
worthy cape
#

I'd reckon that "legitimate" overlays like Discord and NV/AMD ones are likely more recognized and possibly more allowlisted in whatever detection mechanisms there might be, not necessarily because they're desirable but because they're inevitable.

pliant cipher
#

most likely.

worthy cape
#

In any way, the proper venue to get a formal "no" is to mail support@. That's what I did for my hideout editor.

pliant cipher
#

Will do! I'll probably shoot them an email just to see

worthy cape
#

(that tool only wanted to touch the game data, not the client itself)

pliant cipher
#

Did you get a no?

worthy cape
#

Indeed. Outlined the reasoning for my file access, the purpose of the tool, and clarified that it ran completely apart from the game.
Got a well-deserved but friendly and apologetic rejection.

#

(I needed to read data from the pack for decoration definitions, hideout terrain and other geometry)

#

Since then I'm working off a private network store of data so the tool runs on any connected computer without ever touching a game installation, but it kind of precludes ever distributing the tool.

pliant cipher
#

That sounds cool, but yeah I understand GGG not wanting that

worthy cape
#

The only real negative effect that the tool could possibly have had was file locking around patching and technically interfering with the game's own loading if the file was active in another process.

#

I totally understand their reasoning, was just a bit sad after the effort spent developing it.

#

These days it's mostly a learning opportunity to dig into the file formats and figure out how to write a proper game runtime around the constraints of the data.

#

Like the aforementioned I/O system ^_^

pliant cipher
#

I’m coming from the evil online world where stuff like ISBoxer is β€œallowed” but only sort of… tolerated by the devs. CCP never allowed macros like POE does, I’d have KILLED to have some of the macro stuff poe has in that game lol

#

Eve Online*

#

When I saw poe had all these tools I lost my mind

oak estuary
#

I don't think it's as much as macros being "allowed", it's more that there isn't really a way to detect them. GGG could shut a lot of them off by restricting what gets logged to Client.txt or disabling the Ctrl+C functionality

pliant cipher
#

I don’t know how GGG operates but I can guarantee you they know

#

Without a doubt

#

But they do know which tools are popular and such

rigid oxide
#

yeah eve is in a different ballpark, they literally have code to execute any python script on command from the server
afaik ggg doesn't do things like that

pliant cipher
#

I mean yeah but it’s the stance on 3rd party tools that’s the important part, ggg and ccp sound similar, except for macro usage. Poe doesn’t allow multiboxing either but there’s no real need

#

Just spitballing here

#

Btw

#

Just interesting coming from one game to another

grave wren
#

A player may not run more than two copies of Path of Exile on the same computer. They may use multiple computer (or virtual machines, begrudgingly, because we can't really stop that) to run the extra clients.

There is no problem with loading six copies of the game (across multiple computers) into the same area for some reason, as long as they are not controlled by the same set of inputs.

There's no restriction on the number of connections from the same IP.

worthy cape
#

IIRC, you may not multiplex input either for those two copies on the same computer, you have to independently steer each one.

grave wren
#

yes thats the second paragraph

worthy cape
#

Well, it's a bit distinct from the first clause about two clients, but yeah.

pliant cipher
#

Oh, that’s interesting

#

The two clients thing is odd not going to lie

grave wren
#

post is old tho so better check out current TOS

pliant cipher
#

Eve doesn’t allow input broadcasting either so that’s pretty standard now

worthy cape
#

Sounds convenient for trading tho if you can have a second client idling.

pliant cipher
#

If I control 6 clients via GeForce Now is that allowed? Lol

#

It’s 6 different computers

grave wren
#

if you dont multiplex your inputs

pliant cipher
#

But it could be on the same hosted pc

grave wren
#

as in you control each instance sequentially

pliant cipher
#

Fun hah

frank drift
#

even people who follow the rules with multiboxing (2 clients per PC) get banned all the time

grave wren
#

i honestly know no one using it

#

its just not really that useful

frank drift
#

the beastfarmers I know have to create new accounts quite often

pliant cipher
#

That happens relatively frequently in Eve too

#

I got scared when I got up to 11 accounts at once with isboxer not going to lie lol

#

And that was without input broadcasting

neon plume
#

Is there any doc on how the trade site query works and what limits it has?

pliant cipher
#

Don’t think so, I looked at this post

pliant cipher
neon plume
#

thanks!

#

sadly that doesn't say what the max complexity is, or how it's calculated

frank drift
#

Novynn probably doesn't want to state something officially

#

I don't know if it's been reverse engineered but it would be subject to change

worthy cape
#

There's been historical discussion here about vague characteristics about queries, but in general there's nothing you can rely on.

#

I don't remember if it's different limits if you're signed in or not too.

frank drift
#

definitely different limits when signed in IME

#

at least as far as live searches

neon plume
#

alright, added trading button, which seems absolutely useless

rapid pagoda
pliant cipher
#

Ah, that definitely makes sense then

worthy cape
#

Good old muling πŸ˜„

orchid light
#

Anyone here know a few things about RePoE data?

#

Im trying to figure out which mods go onto which items. Basically given an item how can i cross reference this with mods? I assume that there is no 2 spawnweights that will correspond to a item per modifier? Meaning a modifier cant have two spawnweights for the same item?

#

I forgot to add that there wont be two weights > 0 for a mod on an item?

#

So essentially only one tag of an item will match into the spawn weights?

# 
      taggedModifiers.map(mod => mod.getWeightForItem(item)).filter(_.size > 1).foreach(println)
#

This gives no output for leatherscale boots

# 
def getWeightForItem(item: BaseItem): List[SpawnWeight] = {
    spawnWeights.filter(_.weight> 0).filter(weight => item.tags.contains(weight.tag))
  }
#

Right but given an item there should be no two weights matching one item?

#

Yeah given an item, no item should ever have more than one spawn weights for a modifier?

#

Interesting, which weight does it use in that case? Any idea?

#

is there any item with both those tags though?

#

I saw yes

#

Both Viridian and Cobalt it seems

#

Ok, that makes sense.

#

ill need to optimise this code in the future if i want to do this calculation thousand times a second but for now its good

#

I assume mods that adds_tags, adds those tags to the crafted item?

#

Right, makes sense, is there any indication on a mod if its influenced?

#

Im yet to look at currencies and what they say

#

Right now i am just trying to get to a system that can chaos spam a belt

#

Right ok, so this references a tag quiver_elder but this doesnt say which influence?

#

tags.json contains a list of tags, nothing more

#

Wondering if that data is handled manually rather than inferred from repoe

#

Yeah also no tags to identify conquerers

#

whats eyrie?

#

I see yes

#

Not familiar with the names besides crusader

#

Those dont specify anything yikes

#

Ok so

#

Given a base item as a leather belt, the weights i get are

unknown.png
#

Which corresponds with CoE's values

#

However for slink boots they do not match up

#

To high

#

checking them now

#

I need to find a way to group mods? From what i can see just using group isnt enough?

#

Mods currently added

#

And the tag that added it

#

This has a spawnweight of 1000 on boots?

#

Maybe this is for Royale items only?

hazy fiber
#

^

#

for PoB we just filter out everything with royale as part of the name iirc

#

its exactly double, with half having High before the number

orchid light
#

Whats High mean?

#

Lmao, got the jist though

#

Right same mod, but one has higher values?

#

Lmao, can i just filter them out?

orchid light
#

Ye sec

#

Let me do weights by group

#

Right, let me find out why

# 
  {
        "tag": "default",
        "weight": 1000
      }
#

Does seem to have a default weight?

#

So if two tags, one boots one default and boots: 0 then boots spawn weight?

#

Well, it overrides the default correct?

#

This is abit confusing, since the system has inclusive and exclusive logic in one data set

#

Ok, that makes alot of sense

#

Thats also much easier to do

#

wait, how does it find the first tag for a list of tags?

# 
"tags": [
"not_for_sale",
"belt",
"default"
]```
#

leather belt tags

#

does it cross match and find the first to match from both lists?

#

So iterate spawnweight and find the first that matches the tags?

#

Thats it, got it

# 
def getModifiersForItem(item: BaseItem, generationTypes: List[GenerationType.Value] = List(PREFIX, SUFFIX)): List[SpawnableModifier] = {
    ModifierService
      .getData
      .filter(modifier =>
        modifier.domain == item.domain &&
          generationTypes.contains(modifier.generationType) &&
          modifier.getAllTags.exists(tag => item.tags.contains(tag))
      ).map { modifier =>
      val spawnWeight = modifier.getWeightForItem(item)
      SpawnableModifier(modifier, spawnWeight)
    }.filter(_.spawnWeight.weight > 0)
  }
# 
def getWeightForItem(item: BaseItem): SpawnWeight = {
    spawnWeights.find(weight => item.tags.contains(weight.tag)).get
  }
#

Now to build a way we can roll this based on random ness

#

I also assume the required level on mods is ilvl not player level right? πŸ˜›

carmine merlin
#

@orchid light adjudicator = warlord, eyrie = redeemer and basilisk = hunter fyi

orchid light
#

Thnx for all your help, i appreciate it alot

orchid light
velvet fog
#

No, this is a sequence comparison, the first match and return

orchid light
simple ravine
#

I had one of those projects I was playing around with a few years ago, using spawn weights and different crafting methods. It was pretty fun.

#

Then I realized, that it's probably more efficient to just use math than actually bruteforce generate

orchid light
orchid light
simple ravine
#

Yeah, I'm trying to find my thing as well, I was toying around with alt-regal-exa etc

#

nowadays u have a lot more options of course

orchid light
#

Yeah, i do not even know if this is viable

#

This is why i am benchmarking everything as i go

#

Scala might not even be the best option, but as far as parallel goes its the fastest language i know

simple ravine
#

Functional languages tend to suffer a little when it comes to performance, IIRC, but not sure how much in this case

#

I did mine in C#, and nowadays it is a lot more faster ways of doing things there as well

#

Let me see if I still have the code

orchid light
#

However when i do this I will be using a mutation system, so essentially i will be mutating

simple ravine
#

a "while" ago

orchid light
#

I might, MIGHT just swap to Bun πŸ˜›

#

if its faster than Rust like they say....

orchid light
# simple ravine

C# supports the actor framework i am intending to use, so if i can get some speed there i might just use it

simple ravine
#

Eh this is quite more than I remembered πŸ˜‚

#

Maybe something for a rainy night

simple ravine
orchid light
simple ravine
#

Same concept

#

Distributed actor framework

orchid light
#

Well this all depends on the load needed

simple ravine
#

What's the end goal here?

orchid light
#

Mmhmm, its a secret πŸ˜›

#

Im trying to reverse engineer crafting

simple ravine
#

Well the traditional orb crafting is somewhat already documented, no?

#

Bits and pieces confirmed over the years

orchid light
#

Well lets put it this way, Given a crafted item, what is the recipe

simple ravine
#

You can't really reverse engineer something by implementing a novel logic, or... am I missing something?

#

Ah, what is the most efficient recipe I would presume

orchid light
#

Efficient vs cost eventually

simple ravine
#

I think that was my goal as well, and realized that a mathematical approach would be beneficial

orchid light
#

What do you consider a mathematical approach?

simple ravine
#

probability

orchid light
#

Sure, you do not brute force the crafts? only use "if a craft can land a mod"?

simple ravine
#

Well, let's assume alt+regal as a simple option

#

probability of hitting desired_mod_1 and then hit desired_mod_2 would "simply" be like "school math", i.e. hitting 6 on a die, and then hit 6 again

#

which would be (1/6) * (1/6)

#

or something like that

orchid light
#

Well, not exactly but i get the gist

simple ravine
#

what's not exactly? my math, or the logic?

orchid light
#

Well, you need to take into account, affix counts

#

Also, how do you math, "across" crafts?

simple ravine
#

maybe i'm not thinking all the way here, but wouldn't that be possible to factor in?

orchid light
#

It would of course be able to yes

#

I was thinking of starting with, given item + craft, can i roll desired mod, rather than attempt to roll desired mod

#

Than stack a probability of success onto each step

simple ravine
#

hmm yes, there are some mods that will apply weight to coming mods

#

that'd be hard to "math in"

orchid light
#

I mean, not really, just adjust the weights, and calc probablity

#

But a true recipe has, repeating steps and all those things

#

aswell as restart steps

simple ravine
#

you absolutely have mods that will affect the weight of coming mods

#

had at least before

orchid light
#

Such as?

#

Suffixes cannot be changed?

formal kraken
#

cannot roll attack/caster?

orchid light
#

Yes, thats why the input to the crafting method is (item, targetItem)

#

Meaning you can respect metamods in the crafting methods

#

Most of my time in calc is spent in rolling a mod right now, i need a more efficient way of doing it

simple ravine
#

mods add tags, which change the weight of the next mod generation

orchid light
#

Right, so, for handling that, i handle tags added and refetch all modifiers for an item after every craft

simple ravine
#

right, and that is what i meant would be a real hassle with my idea of calculating everything at once

orchid light
#

Yeah, you cannot calc everything at once, you need to iterate until you have a complete item

orchid light
#

almost 7 times faster XD

#

crafting still takes .5ms, so im looking to optimise the modifier lookups now

#

I might just mak a bunch of maps

#

use O(n) lookups

orchid light
fringe belfry
#

anyone down for maps

simple ravine
orchid light
velvet fog
frank drift
#

nice!

slim pilot
past tartan
#

hey, is this the chat for tool developers?

#

i have kind of a niche question, its not even related to poe, but i know you guys know a lot about AHK. so im developing a tool for another game and im in kind of a testing phase and im doing a lot of small changes pretty much every hour/day and my friends who are testing my tool need to download a new release every time they find a bug, is there a way to make my ahk script auto-update the whole directory with all gui pics and additional secondary scripts that my main one runs. so far i have made it so it urldownoadtofile from a raw pastebin page and it works for the main one, but if i do any pic changes to the gui, they have to download the whole zip archive. so sorry to take your time for another game, but in the oficial ahk discord they turn me down because its a game tool....

civic crane
#

i know you guys know a lot about AHK
very bold assumption

velvet fog
pliant cipher
#

The best place to put my info if I’m querying the trade API via a custom tool is in the user-agent header right? In case for some reason GGG needs to contact for something like misuse

mortal bone
#

I typically include a username I can be reached at in the target. Program v1.0 by <username>

pliant cipher
#

I’m going to add my email and username after the program info stuff and before I open source it

#

Ty

earnest goblet
#

Once in a while I have been seeing mentions of people wanting to use Acquisition (which I am maintaining in this fork: https://github.com/testpushpleaseignore/acquisition), but they assume that the project is dead. Seeing that none of the contributors to the original repo are replying to any Issues, and that they haven't committed since 2019, wasn't sure if we could direct people to my fork, or if I'm required to only do a pull request to the original repo?

mortal bone
#

You can always ask @chrome topaz

waxen musk
#

does anyone know what determines special group backgrounds on atlas passive tree?

worthy cape
#

The central art for groups has been called "mastery" since before the advent of actual masteries on the passive tree. Do you mean something different from that?

#

Not sure where the rest of the flair comes from, been a while since I looked at PoB rendering f.ex.

waxen musk
#

i mean this background

unknown.png