#tooldev-general

1 messages · Page 138 of 1

rapid pagoda
#

funny how the belt rewards are so good they make other jewellery not worth gambling on 😐

frank drift
#

Chris loves belts

#

I guess it would be nice for jewels too, so you can focus only on viridians

#

greater/lesser you generally have so much of it doesn't matter

frank drift
#

the ordering is a bit odd... by value or something?

#

I mean lesser > greater > exceptional > grand

#

ah

#

I'd personally go descending from exceptional, but I suppose the game does go in ascending

cunning mantle
#

Is there an alternative to mapwatch? It's not working with the newer maps for me

crimson mountain
#

Is there any place where I can get an XML/CSV/Whatever format for the atlas passive tree? I'd like to construct a "shortest path through X specific nodes" kind of thing, but getting the data seems difficult

hazy fiber
#

you can just take the json off the webpage

civic crane
#

pinned message

hazy fiber
#

oh yeah, or github

crimson mountain
#

Ohh, nice, I completely missed that. Thank you!

crimson mountain
#

Hi, quick question, I'm trying to understand the data structures in the Atlas Passive Tree .json, and one of the ones I don't fully understand is Orbit and Orbit Index. From what I do understand, the Orbit value just determines which "ring" around a given group the node will occupy, 0 being dead center, 1 being the first ring, 2 being the second, so on. No clue as to the radius of these orbits, but I'll see that later. Orbit Index on the other hand is plain confusing. The values range from 0 to 68, and some values are skipped. Any ideas as to how this is handled? It doesn't seem to be degrees or radians, so any pointers would be appreciated, thank you!
Btw all the Orbit Index Values are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24, 26, 28, 30, 32, 36, 38, 40, 42, 44, 48, 54, 60, 63, 64, 66, 68

worthy cape
mortal bone
worthy cape
#

Each orbit has a particular number of slots/indices, counting around the orbit arc from some starting point.

#

For some orbits the indices correspond to a table of angles, for some it divides evenly as Emmitt's code shows.

covert perch
#

is there similar accessible json for the atlas itself?

frank drift
covert perch
#

Sad

worthy cape
#

The Atlas structure is mostly contained in AtlasNode.dat64 with some somewhat unresearched representation for the links. Never got around figuring it out when the representation changed the other league (Conquerors or Echoes?).

storm stirrup
#

Hi everyone 🙂 I got a websocket question if anyone is available

#

My main problem is that I send over a request to connect to a livesearch websocket and I get a HTTP 200 response instead of 101

inland kestrel
#

@velvet fog is it possible to search maps having a certain tag? I know there is /search?q="area_with_water" but i'm looking for an easier way to get a deadlisting of specifically maps.

velvet fog
simple ravine
#

Trade site is not happy right now

worthy cape
#

load: 17.95s Ohno

simple ravine
#

got some 502s and timeouts as well

worthy cape
#

Website heard it was Ascension Day and wanted in on getting off this earth.

simple ravine
worthy cape
simple ravine
#

rip

frank drift
#

trade site dying seems to coincide with console patches

#

could just be coincidence though

simple ravine
#

Too bad, it probably affects player retention somewhat

#

Anyone know if there's any good way of "copying an item to clipboard" on poe.trade?

golden bane
#

I just get timeouts after about 20s. Not able to load search results at all

frank drift
#

live search is working for me, at least enough that I'm seeing a decent amount of items, even though normal search isn't

oak estuary
simple ravine
#

not the official site

#

as it's borked

oak estuary
#

Oh, a search just worked for me when I confirmed the button was there

#

I was going to delete my message once I saw you said poe.trade, but you had already seen it, heh

earnest goblet
#

Wasn't sure if anyone else was using Acquisition, but I just released a few more changes to help get it fixed up. Wasn't sure if there were any major fixes/enhancements that anyone had in mind?
https://github.com/testpushpleaseignore/acquisition/releases/latest

GitHub
Release v0.9.1 · testpushpleaseignore/acquisition

Fixed the "Type" search filter, so that items will fall into categories defined by the data in the below repo:
https://github.com/brather1ng/RePoE
Also fixed the Acquisition upgrade data ...

snow whale
#

i'm assuming they are and are just obscured by the new URL encoding scheme. but I'm not sure how I would go about finding where this one went?

worthy cape
#

Not sure if all files are rehosted under the new scheme and even if they were, you'd have to have a genuine link from GGG to access them as they're intentionally unguessable.

#

If you have access to local game files, consult Art/UIImages1.txt for a sprite sheet telling you which texture you need to decompress and slice to obtain the file you want:

"Art/2DArt/UIImages/InGame/TutorialPictures/Navigation1" "Art/Textures/Interface/2D/2DArt_UIImages_InGame_TutorialPictures_15.dds" 0 0 1337 827
#

They're still part of the game files as of the 3.18.0 set I'm looking at here.

snow whale
#

that is helpful thanks!

frank drift
#

what zao said, GGG does not want you pulling assets from the web cdn

exotic egret
#

All files should be rehosted, but yeah you can't access them without the right hash, which changes from time to time

#

There might still be a permalink you can use, but I won't be able to find it til Monday

simple ravine
#

I mean, pulling assets is what a CDN is for 😇

hazy fiber
#

yeah pulling their assets for their uses, not for randos to keep using

formal kraken
#

but that's probably not the case 🤔

simple ravine
#

hmm, how about make a CDN with their CDN as upstream source

#

I guess the question comes down to "ok, who is putting their credit card in?"

frank drift
#

wouldn't be surprised if multiple high traffic sites were hotlinking images and other naughty things

chrome topaz
#

😳

long sky
velvet fog
#

Want to identify the range of jewels in skilltree-export/data.json (800), but seems to need to scale the x, y of the group. what's the ratio?

worthy cape
#

1.2x, IIRC?

#

(only vague memories of PoB discussions, better verify in which direction it goes and if that's the value)

formal kraken
# velvet fog Want to identify the range of jewels in skilltree-export/data.json (800), but se...

not sure if that helps, but using that radius on the sprite works:

        const j = item?.jewel;
        if (!j || !j.radius) {
            return null;
        }

        const c = new Container();

        let texId1 = `JewelCircle1`;
        let texId2 = `JewelCircle1Inverse`;
        let radius2 = j.radiusMin;
        if (j.radiusVisual) {
            texId1 = `${j.radiusVisual || ''}JewelCircle1`;
            texId2 = `${j.radiusVisual || ''}JewelCircle2`;
            radius2 = j.radius;
        }

        const sprite1 = xr.createSprite(texId1, false);
        sprite1.width = 2 * j.radius;
        sprite1.height = 2 * j.radius;
        sprite1.blendMode = BLEND_MODES.ADD;
        c.addChild(sprite1);

        // render the second (inner) circle, either when there is a radiusMin or it is a 'visual' circle
        if (radius2) {
            const sprite2 = xr.createSprite(texId2, false);
            sprite2.width = 2 * radius2;
            sprite2.height = 2 * radius2;
            sprite2.blendMode = BLEND_MODES.ADD;
            c.addChild(sprite2);
        }
#

so you can probably get the accurate-ish values from combining the radius with the sprite circle radius?

velvet fog
#

calculated the distance between Imbalanced Guard and Prismatic Skin to be 939, 1.2x works for small radius(800)

#

seems work, thanks!

velvet fog
grave wren
#

Interesting question does anyone know something for him? Never heard of anything like it

formal kraken
#

this is crazy how do you even socket a skill blindly, does the game have any support for that?

hazy fiber
#

I suggested just navigating the tree json itself as probs the easiest way to do it

#

I cant think of how else you would go about it

grave wren
#

Don't think so

worthy cape
# 
You are standing on a small int passive. There are three nodes nearby.
* 1) small int passive
* 2) small int passive
* 3) Elemental Equilibrium keystone.
You can hear a jewel slot humming nearby. You are likely to be eaten by a grue.
hazy fiber
#

maybe not to difficult to socket gems on controller blind

grave wren
#

I thought a bit about it and the best I could think of was just routing to the next notable or keystones

#

Zao release when

hazy fiber
worthy cape
#

I forget, does any of the clients have preset build planning integrated kind of like you can pre-allocate nodes in the Sentinel pane now?

#

Got vague memories of archetypes in the China client?

grave wren
#

Yeah I think Chinese realm has it

#

I honestly like the adventure style mockup and with more Infos like nearby keystones that might be the best for blind players

#

Maybe even just a tool to inspect trees of builds at first

hazy fiber
#

dont think it lets you pre-allocate, just displays what you have left to grab vs planed tree iirc

long sky
worthy cape
#

^_^

hazy fiber
#

well he responded to my suggestion, so hopefully he figures somthing out

grave wren
rapid pagoda
violet path
#

historic

frank drift
#

woah

velvet fog
#

wow

simple ravine
#

that bug has annoyed me for years indeed

ebon oasis
#

i really wanna know what it was

frank drift
#

it only happens with windowed or fullscreen windowed right?

#

my best theory is that the window size gets read when tabbed out/minimized, giving an invalid value, and then that value is used somewhere in text spacing calculations

worthy cape
#

I've got a bit of an idea of what it is, but not gonna spoil the fun 🙂

#

Definitely didn't guess the particular failure mode over the years.

worthy cape
#

One of the workarounds has been to slightly resize the window however to make it adjust the font size and produce fresh text.

velvet fog
frank drift
velvet fog
#

ya, use ctrl+alt+c to copy item from game and ctrl+v paste

velvet fog
grave wren
#

awesome thanks ❤️

velvet fog
#

Most of the data is from a royale hobby group in China, 8 are datamining

#

Some of the unique may have incorrect required level

grave wren
#

Interesting, does china still have royale active?

velvet fog
#

Ya, they send me the data and wish me to put it on web

grave wren
#

very cool 🙂

primal hazel
#

Hey quick question. I have tried queries to the trade site recently and have been getting 1020 access denied errors. I had no issues before this league. I'm waiting for a response from GGG but I was wondering if someone else has a solution

formal kraken
#

so maybe you're on the bad list of cloudflare or some rate limiting/tool detection/firewall rule is happening

primal hazel
formal kraken
#

User-Agent: OAuth {$clientId}/{$version} (contact: {$contact}) ...

#

I can't remember if it was a 1020 though, I just remember it was when exchanging the access token

primal hazel
#

Yea I followed that and still get the same errors. I've also tried it on Postman and still the same.

#

But I can use the trade site no issues lol

dawn thunder
#

working on overlay docked to the mouse for debuffs...

unknown.png
dawn thunder
#

Python Script for that...
no Gui Stuff Added yet...
and while loop runs every frame but for some people who want to check it out

dawn thunder
#

Ok started on ui Framework...
And need a while...
Stupid question is there a resource File we can Access for Icons Like debuffs ?
As i pulled all of them from the wiki

frank drift
bronze kestrel
#

Are the people who run poe.ninja in this channel?

hazy fiber
#

rasmuskl? hes around sometimes

bronze kestrel
#

Was gonna ask if they have an API they can grant us access to for the wiki discord. I’m reviving our wiki bot (full rewrite) and wanted to see if there’s something by way of a json API provided

hazy fiber
#

@fickle yew ^ you are wanted

bronze kestrel
#

@hazy fiber I think my brain didn't realise I could have @'ed them.. derp.. Need more coffee this morn

frank drift
bronze kestrel
#

Yeah I vaguely remember it. But at the same time -- I could get why a public API may not be available. I could just use the query param builder to issue those and return the resulting link, but frankly it would be so much nicer to get some JSON and construct nice embeds/interactions using the discord API for people to use.

#

Running own git project is so crisp though. Makes me feel so good about things all being as I like it. Issues nicely written out, projects buzzing along (new project views are pretty nice), CI built and humming. Just zen all around. peepoBlanket

fickle yew
#

@bronze kestrel Swagger page died in a major version upgrade. The API still exists. Feel free to DM me.

frank drift
#

pointers, should've known

worthy cape
#

Leaky caches, yay.

#

It was interestingly enough never on my mental map that it could be stale data, I was dead-set on some sort of clobbering or race.

#

Guess I'll now have to put another bug up on the motivational list of "join the company just to fix the bug" reasons.

dim stream
#

anyone super smart in PoB know how to put into custom modifiers a way to include Forbidden Rite's self damage on cast?

#

Nevermind, just giving myself 200% increased cast speed to trick PoB into thinking I have 5 casts per seconds

carmine merlin
#

ignore the life after part, that was just my testing

dim stream
vestal pond
#

For BlightTopologyNodes.dat(64) and DelveFeatures.dat(64)
I found there is a column with type: array|i32 which pointed to a data section with Offset + Length < Offset-Of-Next-Data-Section
And all of their missing length are equal to their own length

And after some test I think there should be a data type like tuple which read a pair of data each time
The following is an example of three types I tested separately

  • array|i32
  • array|i64
  • array|tuple|i32
unknown.png unknown.png unknown.png
#

And also BlightTopologyNodes

unknown.png unknown.png
rapid pagoda
worthy cape
#

Been a while since I looked at it but I think that PoB exporter has some sort of min-max pair like concept as a type.

oak estuary
past merlin
#

New tool, fresh off the press! This one's kinda niche, but here we go: I recently learned that some people gave up on using online filters due to issues with their caching, that prevent frequent filter adjustments on FilterBlade from being reloaded in-game until a few minutes (sometimes more) have passed. So those users are falling back to offline filter management, but that means downloading, copying and possibly renaming their filter from <downloads> to <poe filters> every time.

So for those that gave up on that and are just not using loot filters anymore, I'm happy to present filtersnatch! It's built to support the offline filter replacement workflow by watching your download dir and automatically promoting any new filter file you download to your poe filters directory, renaming it as well. It's written in Go, runs in the tray (except for initial setup, where you tell it how you'd like it to do its thing) and, well, that's it. It's really not a very useful tool.

But I'm still hoping you'll check it out - https://github.com/omriharel/filtersnatch. It turned out pretty neat IMO. Eagerly awaiting any feedback you may have!

gilded tree
past merlin
worthy cape
#

On the act of tabbing, that's a bit sketchy as it interprets changing focus as an explicit user action. If it was an explicit hotkey like APT's /hideout it'd have precedent of being "fine".

past merlin
#

That's a great point. Then yeah, I'll look into adding an explicit configurable hotkey to trigger /itemfilter - with the one caveat being that I don't want to rely on AHK. I want to make it as easy as possible to get this tool working on platforms other than Windows down the line

formal kraken
#

What if it is bound to tab /s

gilded tree
velvet fog
carmine merlin
#

Ooh nice

#

Does that work the same for ailment threshold?

#

Oh realised you don't show ailment threshold as a number

velvet fog
#

don't know how to calculate the ailment threshold

#

Shock and Chill magnitudes is now: 1/2*[Damage % of Ailment Threshold (see below) ]^0.4

hazy fiber
#

ailment threshold is a different percentage for different bosses, and I dont think we know any of them atm right?

#

I think we have an idea for either shaper or sirus based on patchnotes and testing, and PoB uses that for all bosses iirc

carmine merlin
#

You have this for ailment threshold

unknown.png
#

on shaper

#

Thought it was the same calc as life but switch life% to ailment threshold%

carmine merlin
velvet fog
carmine merlin
#

What's the 0.47 "Map Boss Multiplier" you have on the end there?

velvet fog
#

it's from MonsterMapBossDifficulty.dat, last column

carmine merlin
#

Is that only meant to apply to Ailment Threshold and not life?

velvet fog
#

life have another column in this file

carmine merlin
#

I don't think the ailment threshold includes that 0.47 multiplier. The numbers look too low

velvet fog
#

but this value matched our test

#

Brittle magnitude=25 * (Damage / ailment threshold)^0.4
ailment threshold = Damage/((Brittle magnitude/25)^2.5)

#

ailment threshold = 2.32*0.76/((15/25)^2.5) = 6.32

carmine merlin
#

Do you have any increased effect of ailments?

velvet fog
#

this test was done in patch 3.15 and did not increase effect of ailments. What do you think this number should be?

carmine merlin
#

I'm trying to do some tests right now, your number seems to be correct from what data I've gathered so far

carmine merlin
#

I just did a Shaper run and from looking at the numbers, the ailment threshold might be even lower unless there's something wrong with my builds PoB

#

I somehow managed to inflict a 25% scorch on shaper with my 1 link flame dash

#

There's no way my 1 link flame dash hit for 770k, I'm so puzzled right now

velvet fog
#

ya, we tested Scourch, about 3.97m ailment threshold, much lower than formula

carmine merlin
#

Putting 4m ailment threshold fixes it, wonder if they changed the formula at some point

formal kraken
#

is it possible ailment thresholds are different for different ailments?

carmine merlin
#

They have different formulas for each

formal kraken
#

ah

velvet fog
#

Will they change the formula but forget to write the patch notes?

carmine merlin
#

Maybe spaghetti code?
I'm also getting a lot higher brittles than I would expect with 7mil ailment treshold

#

Getting 5.7% brittle with a level 1 frostbolt

#

Like shaper has 1.7mil ailment threshold for brittle

velvet fog
carmine merlin
#

Yeah might be a good idea

velvet fog
tawny agate
#

Don't know if the right people are here but the wiki's DB is agonising

worthy cape
velvet fog
# 
MediaWiki internal error.

Original exception: [8d3de9c0ef15c75e9dee94c8] 2022-06-05 12:42:21: Fatal exception of type "Wikimedia\Rdbms\DBQueryDisconnectedError"

Exception caught inside exception handler.

Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.
#

got this error on random page, search broken too

tawny agate
ebon oasis
#

is it just me or does trade break when you search for like
gloves + accuracy per frenzy + something else?

grave wren
#

broke for me when adding a max chaos value

frank drift
#

interesting

kind otter
#

trade brakes for me when i add more then 2 values

stoic mason
#

Trade is just broken right now in general

formal kraken
#

I wish I had the budget (time and server costs) to re-implement the trade site with a graphql + elasticsearch backend

simple ravine
frank drift
#

the Norwegian report is being backed by consumer groups in 18 EU counties, good chance the EU as a whole cracks down

simple ravine
#

Personally I think that's a good thing for the whole industry if it becomes forbidden, because it makes other game developers who feel enticed to add this mechanic to stay competitive

#

It's a detrimental mechanic for the young players to be lured into. I can relate more and more to this, as my son is growing up now, and starts playing more "free to play" games. Now asking for Robux constantly.

#

I mean, I probably spent $1k+ on those boxes in PoE lol

frank drift
#

yeah I'd personally love to see more regulation

simple ravine
#

Better spend more time than that, than cracking down more on GDPR stuff that stifles innovation completely

#

(yes, GDPR is of course good in some essence, but it's becoming absurd)

worthy cape
#

There's a reason why Diablo:Immor[t]al didn't launch in the Netherlands and Belgium 😉

simple ravine
#

They've must have thought for themselves "I'm going to hell" when announcing that crap 😂

snow whale
formal kraken
#

for this usecase it is probably overkill since the set of returned values is pretty static (maybe useful for bulk exchange?)

hazy wigeon
lunar crypt
#

Hey guys, if you implemented wiki functionality in your tool, please have a look if you recognize this IP address:

178.62.86.211

#

For the past few days we’ve been experiencing some issues and this address hits a lot of api calls

frank drift
#

digitalocean hmm

long sky
#

This sort of API call

unknown.png
frank drift
#

maybe you should require a user-agent like the trade site 3xG

long sky
#

Unfortunately we're a bit limited with what MediaWiki allows when it comes to their default API functionality 🙃

simple ravine
simple ravine
mortal bone
#

I was just about to say. This is offered in cloudflare for free. Website -> Security -> WAF -> Tools

#

I don't have pro, but I do believe you can do a bit more with managed rules

lunar crypt
#

thanks, got it enabled

simple ravine
#

now let's see the oncoming complaints of things breaking 😄

frank drift
#

the more they know they were being naughty the more likely they are to indignantly complain

lunar crypt
#

This only started a few days ago

#

And seems like it’s scraping or cache busting

hazy wigeon
long sky
#

has anyone built a monster/minion tool that's querying our API?

unknown.png
hazy wigeon
#

looks like its a local user in vietnam, I am not 100% sure anyone here will have info, if you visit that IP it brings you to the ISP portal for a large ISP in vietnam

simple ravine
formal kraken
simple ravine
#

Unfortunate to see blocking of API access, but oh well

worthy cape
frank drift
#

tragedy of the commons

simple ravine
#

Well, it depends on how resource heavy they are, and if it's possible to perhaps optimize something that'll alleviate it

worthy cape
#

As they tend to hit the database with novel queries it puts undue load on whatever dinky backing server is powering it, dodging all the clever caches.

formal kraken
worthy cape
#

They're by definition transient results, good luck caching fairly unconstrained SQL.

#

Wikis are normally lightweight in that normal page use is prepared server-side when changes happen, while the API exposes the underlying data.

#

"cached" in as having things pre-rendered server-side in kind of like a static site setup, not "cached" as in "the database is warm in RAM".

long sky
#

we were getting fucked by bing crawlers hitting a special cargo query page

#

API should be totally fine to use now, but if anyone sees any problems with it please let us know!

mortal bone
#

time for a robots.txt 😄

long sky
#

Just blocked access to that page for non-logged in users

velvet fog
#

how to draw correct circle?

hazy fiber
#

looks good, just some face wrong way

formal kraken
#

if I remember correctly depending on the two angles of the nodes you have to figure out which way to go, clockwise or counter clockwise (basically to decide what is inner and what is outer for the arc)

#

^ if you need an svg

velvet fog
#

thanks

simple ravine
#

Because I'm changing jobs, I'm to 90% relieved of duties at work, so I have plenty of time to do something interesting. Question is if it's worth my time to finally release a production-worthy version of PoeSharp (C# library for parsing poe data)

#

or if I should do something else poe-related.

worthy cape
#

Free agent \o/

simple ravine
#

It's so strange to have so much time on my hands 😂

#

I even had time to level a character to maps, which I haven't done since Legion when I for the first time got a character to lvl 100. I guess it's also due to the game not motivating me too much for a while.

gilded tree
gilded tree
civic crane
#

@velvet fog btw how do you type in chat, seems poe doesnt support IME

hazy wigeon
#

Ive now added support for HasValue and IsTrue meta commands in poefilterx, Im thinking Ill have 3.18.3 out later today with a bunch of bug fixes, better logging, and proper full support for the commands I missed that the wiki didnt list but official PoE website has

#

HasValue will prevent the whole block from being written if whatever comes after it is null/empty, handy for validating against injected variables

#

IsTrue is basically the same but it explicitly just expects whatever comes next to be True or False

#

As an example:

{
    HasValue %MirrorTier_Scarab%;
    BaseType == %MirrorTier_Scarab%;
    Style mirror-tier;
}
{
    HasValue %ExaltTier_Scarab%;
    BaseType == %ExaltTier_Scarab%;
    Style exalt-tier;
    {
        StackSize > 1;
        Style mirror-tier;
    }
}
#

If we remove the HasValue call we would get stuff like:

Show
SetBackgroundColor 46 146 248 255
SetFontSize 45
PlayAlertSound 6 300
PlayEffect Cyan
Which technically is correct. There was no BaseTypes so the BaseType line got wiped, but everything else stayed, which is not what we want of course. But with HasValue that block doesnt get rendered at all

simple ravine
#

Thing is that I normally have too big ambition to follow it through to production ready stuff

#

Like, PoeTradeMacro/Awakened Poe Trade like app with trade stuff included

#

And some no-code/low-code interface for high customizability, but then, yeah, stuff grows out of hand 😄

#

much like PoeSharp, which is in its 3rd or 4th incarnation since 2017-2018

frank drift
#

easy one day project

stuck blaze
#

Hello

#

I'm trying to consume the public stash tab API but I don't think I can catch up. I'm grabbing the latest next_change_id from poe-ninja. But it takes me about 2-3 seconds to download one request. And one request contain ~100 stashes. But according to poe ninja's stash_tabs_processed it increases like 100 stashes per second. So I'm at about 50% speed of being able to catch up. Any suggestions?

civic crane
#

did you make optimization to take next_change_id from response early, ie streaming

stuck blaze
#

Yes

# 
                {
                    using (var contentStream = await result.Content.ReadAsStreamAsync())
                    {
                        var api = await JsonSerializer.DeserializeAsync<API>(contentStream);
                        objects.Enqueue(api);
                        next_change_id = api.next_change_id;
                    }
                }```
#

dont mind DeserializeAsync it takes like 50 ms so its negligable

civic crane
#

not familiar with json in c#, but from quick docs it seems that it waits for full object shrugR
anyway if I'm mistaken
you will better understand what's happening if you measure time to first byte, time from TTFB to make next request.

worthy cape
#

Does that do the "read just a few bytes to grab the ID and defer the read of the rest" thing?

#

You can rely on that the first fistful of bytes are {"next_change_id":"3001-5259-4585-5452-1849",... and issue another request while you're reading the first one off the network.

stuck blaze
#

Thanks I will investigate streams further

#

huge ttfb 😐

civic crane
#

yeah, it's ok for more older ids, when you catch up it will be ~200ms (depends on your ping to US, where api is hosted)

stuck blaze
#

I'm grabbing the latest id from poe-ninja so I dont think it's that old.

civic crane
#

by the time you paste it, it's already old by 2-4 fetches

simple ravine
#

According to my tests, @stuck blaze it's not really worth getting the id early, as the download and parse takes trivial amount of time compared to TTFB

civic crane
simple ravine
#

I guess it depends on where you download from and your bandwidth

#

for me the download was very fast

#

but I got a 1000 mbps connection

stuck blaze
#

im from sweden, I got 1 gigabit

simple ravine
#

same here, sweden gbps

stuck blaze
#

can u show ur ttfb?

simple ravine
#

TTFB is as slow for everyone

stuck blaze
#

how do you catchup to the river @simple ravine ?

simple ravine
#

have to dig the code up, was a while ago I was experimenting with the river

#

but...

objects.Enqueue(api);

is that processed on the same thread?

stuck blaze
#

negative

# 
_ = Task.Run(async () =>
    {
        try
        {
            HttpResponseMessage ninjaResponse = await ninjaClient.GetAsync("api/data/GetStats");
            ninjaResponse.EnsureSuccessStatusCode();
            string ninjaBody = await ninjaResponse.Content.ReadAsStringAsync();
            var ninjaObject = JsonSerializer.Deserialize<NinjaResponse>(ninjaBody);
            var next_change_id = ninjaObject.NextChangeId;
            Stopwatch sw = new Stopwatch();
            while (objects.Count < 10)
            {
                sw.Restart();
                using (var result = await poeClient.GetAsync($"public-stash-tabs?id={next_change_id}", HttpCompletionOption.ResponseHeadersRead))
                {
                    using (var contentStream = await result.Content.ReadAsStreamAsync())
                    {
                        var api = await JsonSerializer.DeserializeAsync<API>(contentStream);
                        objects.Enqueue(api);
                        next_change_id = api.next_change_id;
                    }
                }
                Console.WriteLine($"download and parse: {sw.ElapsedMilliseconds}");
            }
        }
        catch (Exception e) { Console.WriteLine(e.Message); }
        finally { Console.WriteLine("I died"); }
    });

    while (true)
    {
        if (objects.TryDequeue(out API root))
        {
            Console.WriteLine($"stash count: {root.stashes.Count}");
            foreach (var stash in root.stashes)
            {
                if (stash.league == null || !stash.league.Contains("Sentinel"))
                {
                    continue;
                }
                Task.Run(() => logic.AddStash(stash));
            }
            stashCounter += root.stashes.Count;
            Console.WriteLine($"{stashCounter / (masterStopwatch.ElapsedMilliseconds * 0.001):N2} stashes/sec\n");
        }
    }
simple ravine
#

digging thru my code

#

it's eh, from 2020, .net 6 rc1 code

#

updating some stuff

stuck blaze
#

nice 👍

#

regardless I'm very worried that my ttfb is about 1.5-2s and contain about 100 stashes. And according to poe-ninja the river grows with about 100 stashes each sec. 😐

#

so if processing took 0 ms I would still never catch up to the river

#

crazy how ttfb is so high when my ping to the US is around 100-200 ms

simple ravine
#

not sure if rasmus is still using white listed access, which you and I would be 60 seconds behind as well

stuck blaze
#

im gonna make something similar to poe-ninja with price history so a 60 second delay wont matter much to me

#

but I need to be able to keep up with river

#

or data would get old 😦

#

who is rasmus?

#

😂

simple ravine
#

poe.ninja developer

#

looks like I am not catching up yet either

#

maybe have to take a second look at it

#

bear in mind I haven't touched this much since 2021

stuck blaze
#

cool thanks! I'm gonna try it now

simple ravine
#

@stuck blaze , so if you go to Fetcher.cs, you'll see a Background service being run

stuck blaze
#

yeah I already tried it

simple ravine
#

on row 60-66 u can see the flow

stuck blaze
#

little to no difference

simple ravine
#

where _ = GetChanges(response, id); is currently a fire-and-forget

#

and writes to a Pipe, so as soon as the next id is available, a new request is sent out

#

hmm, not sure why I wrote it like that, I'm going to simplify it a little

simple ravine
#

@stuck blaze - you catch up after a while... like 10 minutes of fetching

gilded tree
stuck blaze
#

I wanna make pretty graphs with data from today/this hour I really don't want it more then a few minutes behind ideally

gilded tree
#

Yeah sounds good

stuck blaze
hazy wigeon
velvet fog
civic crane
#

I guess it's behind compilation flag that disables it for international client, because I tried to input something and poe was skipping them without any visual indication of compositing and nothing has been pasted after I finished compositing shrugR

velvet fog
#

@compact isle garena taiwan oauth user login had broken after yesterday patch, i had sent you email but not received reply

hazy wigeon
#

So I am looking at the data returned from poe ninja for the HelmetEnchant endpoint, which looks like so for one entry:

{
            "id": 6466,
            "name": "Kinetic Blast has a 75% chance for an additional explosion",
            "icon": "https://web.poecdn.com/gen/image/WzI1LDE0LHsiZiI6IjJESXRlbXMvR2Vtcy9DbHVzdGVyQnVyc3QiLCJ3IjoxLCJoIjoxLCJzY2FsZSI6MX1d/1b7aff0832/ClusterBurst.png",
            "variant": "75",
            "itemClass": 0,
            "sparkline": {
                "data": [
                    0,
                    39.62,
                    87.28,
                    413.28,
                    454.33,
                    460.06,
                    466.93
                ],
                "totalChange": 466.93
            },
            "lowConfidenceSparkline": {
                "data": [
                    0,
                    39.62,
                    87.28,
                    413.28,
                    454.33,
                    460.06,
                    466.93
                ],
                "totalChange": 466.93
            },
            "implicitModifiers": [],
            "explicitModifiers": [],
            "chaosValue": 181.63,
            "exaltedValue": 1.00,
            "count": 99,
            "detailsId": "kinetic-blast-has-a-x1-chance-for-an-additional-explosion-75",
            "tradeInfo": [
                {
                    "mod": "enchant.stat_3105097589",
                    "min": 75,
                    "max": 75
                }
            ],
            "listingCount": 382
        },
#

And Im trying to translate this into what a poe filter is wanting for the HasEnchant condition, but I think what poe filter would want is:

HasEnchant == "Enchantment Kinetic Blast Explosions 2"

#

Is there any way at all to translate between the two?

worthy cape
#

The mod is named EnchantmentKineticBlastExplosions2 in Mods.dat and references the stat of kinetic_blast_%_chance_for_additional_blast in Stats.dat, the stat maps to a name in stat_descriptions.txt as:

    1 kinetic_blast_%_chance_for_additional_blast
    2
        #|99 "Kinetic Blast has a {0}% chance for an additional explosion"
        100|# "Kinetic Blast has an additional explosion"
#

Not sure if one can munge those together to get a meaningful filter identifier, whatever those are.

hazy wigeon
#

Where are "Mods.dat" and "Stats.dat" ?

worthy cape
hazy wigeon
#

But also in that json there, none of the data returned from poe ninja maps to any of that it seems

#

since the poe ninja value is
kinetic-blast-has-a-x1-chance-for-an-additional-explosion-75

worthy cape
#

Not sure where rasmus has conjured up that identifier but it smells like a custom one.

#

The trade stat is the closest to machine-readable, but those are not mappable.

hazy wigeon
#

this part looks the closest to "could maybe mean something" but Im having trouble figuring out whhat it maps to

"tradeInfo": [
                {
                    "mod": "enchant.stat_3105097589",
                    "min": 75,
                    "max": 75
                }
            ],
#

Okay so, in stat_translations (I am using RePoe github for this) we have:

 {
    "English": [
      {
        "condition": [
          {
            "max": 99
          }
        ],
        "format": [
          "#"
        ],
        "index_handlers": [
          []
        ],
        "string": "Kinetic Blast has a {0}% chance for an additional explosion"
      },
      {
        "condition": [
          {
            "min": 100
          }
        ],
        "format": [
          "ignore"
        ],
        "index_handlers": [
          []
        ],
        "string": "Kinetic Blast has an additional explosion"
      }
    ],
    "ids": [
      "kinetic_blast_%_chance_for_additional_blast"
    ]
  },
#

so the english translation seems like what we want

worthy cape
#

I assume some of this may be exposed in RePoE too.

hazy wigeon
#

And then we have

"EnchantmentKineticBlastExplosions2": {
    "adds_tags": [],
    "domain": "item",
    "generation_type": "enchantment",
    "generation_weights": [],
    "grants_effects": [],
    "group": "SkillEnchantment",
    "implicit_tags": [
      "attack"
    ],
    "is_essence_only": false,
    "name": "Enchantment Kinetic Blast Explosions 2",
    "required_level": 75,
    "spawn_weights": [
      {
        "tag": "helmet",
        "weight": 100
      },
      {
        "tag": "default",
        "weight": 0
      }
    ],
    "stats": [
      {
        "id": "kinetic_blast_%_chance_for_additional_blast",
        "max": 75,
        "min": 75
      }
    ],
    "type": "EnchantmentKineticBlastExplosions"
  },
#

yeh

#

So it seems like I just gotta figure out the pattern for translating:

kinetic-blast-has-a-x1-chance-for-an-additional-explosion-75
into
kinetic_blast_%_chance_for_additional_blast

worthy cape
#

I have never seen that type of identifier before, pretty sure it's a @fickle yew special.

hazy wigeon
#

but I think its actually first being translated into
Kinetic Blast has a {0}% chance for an additional explosion

hazy wigeon
#

Lets look at some others here...

#

tornado-shot-fires-an-additional-secondary-projectile-0
Maps to...
Tornado Shot fires an additional secondary Projectile

#

well thats handy

#

I think I got it, so the x1 in the name is {0}, and the last number is the "args" to pass backwards in

#

So like
x1-foo-x2-10-20 I guess would be actually
10-foo-20

#

So we have this one unfortunately, guess I was wrong
"boneshatter-has-x1-chance-to-grant-x2-trauma-24"
which maps to
"Boneshatter has {0}% chance to grant +1 Trauma"

hazy wigeon
#

Success!

unknown.png
#

@earnest radish I sniped your script and converted it to a json object and cited your repo in my code. I'll output a reference in the script as well

#

MIT License club 😎

#

Aight that another basetype down

#

I... think all I have left is to parse Annointments for gold/silver oils

#

And then uniques basetypes

#

Oh hell yeah, perfect!

hazy wigeon
#

@earnest radish just a heads up, looks like you're missing the new abyss unique jewels,
"Amanamu's Gaze", "Tecrod's Gaze", "Ulaman's Gaze", "Kurgal's Gaze",

#

gotcha, makes sense

fickle yew
rare gulch
#

hello my friends, was looking at some old code and i guess some stuff has changed.

var url = https://www.pathofexile.com/character-window/get-stash-items?league=Sentinel&tabs=0&tabIndex=0&accountName=druuu

    var opt = {
      "method": "get",
      "headers": {
        "Cookie": "POESESSID=" + POESESSID,
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36 Edg/102.0.1245.30"
      }
    }
    console.log(opt)
    try {
      response = UrlFetchApp.fetch(url,opt)
``` with a valid possessid and i'm getting 403d
#

In the past it was a mix of an issue with useragent not being in the header and trying to hit http instead of https but now i'm not really sure what is wrong anymore

hazy wigeon
#

I think now its a Bearer: token?

rare gulch
#

isn't that only for the /api/ endpoint?

rare gulch
#

okay I figured it out I think, the code above SHOULD* work, for some reason google app scripts is getting forbiddened every time. If I run the same code locally it works fine, kinda sus but is what it is

rigid oxide
worthy cape
#

UA:s should typically include your product name and version, as well as contact information.
If you're spoofing a browser it may be part of the problem.

#

Not sure if poesessid is IP-bound so if you've generated it on a different host than the one you're running on it might be part of the problem.

rare gulch
#

yeah, back in harvest a lotttt of people used google app scripts/ urlfetchapp to download their crafts, it must of gotten blacklisted after that point. I just snatch it locally and then push it up to a spreadsheet instead. also that makes a lot of sense about the poesessid

wide otter
hazy wigeon
wide otter
wide otter
frank drift
velvet fog
#

@worthy cape I have some free time and would like to restore the model viewer for monsters and unique items, do you have any suggestions?

worthy cape
#

I don’t remember if I’ve written up the documentation for the newer versions of the three mesh formats but the newer versions are all unified around the mLOD inner format of mesh data. My loaders are all using that as the final representation and convert the old versions as good as they can when loading.
At some point in time the encoding for normals/tangents changed from a biased uint8 to a proper SNORM so that needs to be mapped offline as it’s cumbersome to do in shaders.
Material wise I still recommend pulling out the texture maps and using those as textures in a simple shader, I’ve only scratched the surface of a single material myself and even translation into HLSL is hard and full of guesswork.
On animations I have historical docs and the format doesn’t seem to have changed much but I’ve never done the actual pose computations with PoE data.

#

It’s been many weeks now since I touched the viewer as I’ve been stuck trying to get the data download and ingestion pipeline done. Databases and persistent state really isn’t my thing 😄

#

I did a bit of glTF exporting for debug visualization and while it could represent most of the data, it comes very short material-wise if you want much anything past the stock PBR input textures.

worthy cape
#

I’ve been thinking about whether I could do simplified materials for viewer fallback but have not made much progress there, rather opting to go all the way as I’m on a compatible shader API.

velvet fog
velvet fog
#

there are new version 2 and 3 smd format

wide otter
#

but that has the ability to add base types etc

worthy cape
# velvet fog I can't parse this file ("Art/Models/MONSTERS/DoomPersonified/rig_5d5d58b3.smd")...

Yeah, v3 uses mLOD while v1 and v2 have the legacy format. The trailing data in all formats is still much the same.
The state of the art of my parsing is in code has mLOD bits separated into dolm.[ch]pp at https://gitlab.com/zao/poe-cpp/-/tree/squint-prototype/libpoe/poe/format and branching logic and the rest of the formats in {fmt,smd,tgm}.[ch]pp.
See also {fmt-v9,smd,tgm}.bt at https://bitbucket.org/zao/gggtools/src/evolve-ggpkviewer/spec/.

#

My code can parse everything up through PoE 3.17.3, have yet to bring in Sentinel so not sure what surprises lurk there.

worthy cape
#

Some of the things like the more exotic vertex formats may be misinterpreted, but the core of it should be reasonable.

velvet fog
#

Is it ok if I use one of the more vertex sets of mLOD?

worthy cape
#

I'm not quite sure what you're asking?
Each mLOD section has two submeshes where the first one is of higher geometric density than the second, my assumption has been that the second mesh is a low-LoD version of the mesh rather than a shadow-caster or collision mesh as it still has the full vertex format of components.

#

When visualising them they seem to be around half the tri-count of the first mesh but still intended for displaying.

velvet fog
#

I don't quite understand how mLOD works

#

I'll draw it tomorrow and see how it goes

worthy cape
#

The SMD as a whole has four shapes. The mLOD section has two submeshes (here called geoms due to history), the first with more vertices/triangles than the second.
Each submesh has its own vertices and indices and is segmented into shapes inside by a list of index ranges.
Shapes are typically there for material assignment and culling, for SMD the materials are assigned in the SM file.

#

This assigns the first material to the first shape, second to second, and third to shapes 3-4.

version 5
SkinnedMeshData "Art/Models/MONSTERS/DoomPersonified/rig_5d5d58b3.smd"
Materials 3
    "Art/Models/MONSTERS/3.17Monsters/DoomPersonified/Textures/DoomPersonifiedc.mat" 1
    "Art/Models/MONSTERS/3.17Monsters/DoomPersonified/Textures/eyec.mat" 1
    "Art/Models/MONSTERS/3.17Monsters/DoomPersonified/Textures/DoomPersonifiedc.mat" 2
BoundingBox -113.512 -51.7014 -230.355 113.512 46.6893 -0.060627
#

If you're drawing without materials, you can typically draw the whole submesh in one call, while if you use materials or toggle shapes on/off, you would bind the submesh geometry data and draw ranges of indices.

#

Some meshes have multiple overlapping shapes that can be toggled on/off, this is sometimes done when equipping gear so that the underlying shape doesn't clip through the equipment.

#

The stride (size of each vertex) of the vertices is determined by the vertex format, here 60:

case 60:
    ret->format = (VertexComponent)(VtxPosition_3f32 | VtxNormal_4i8 | VtxTangent_4i8 | VtxTexcoord0_2f16 | VtxSkinBones_4u8 | VtxSkinWeights_4u8);
worthy cape
#

It seems like my Git client didn't tell me about the state of the working copy in time, I've got plenty of changes to the 010 editor templates that I haven't committed yet, so the C++ source is probably most accurate.

lusty musk
lusty musk
#

I have parsed the HTML of poedb one too many times 😛

wise copper
lusty musk
#

with poedat I can load tables into duckdb using 20 lines of python

lusty musk
worthy cape
#

(Windows, Linux, macOS)

lusty musk
worthy cape
#

You're going to have a Bad Day on that platform unless you've got Rosetta, considering that the reverse engineered Oodle decompressors are x64-only.

lusty musk
#

I guess for this project I don't technically need the baseitemtypes.dat

#

it was just for cross-referencing blight oil names but I can deduce that

wise copper
#

Lol... If I remember, I can just generate it and send you a file in like 7 hours

lusty musk
#

nah I got it

#

12 = silver oil, 13 = golden oil

#

all I care about

#

thanks though

wise copper
lusty musk
# 
dbcon = duckdb.connect(database=':memory:')
load_poedat_as_table(dbcon, "BlightCraftingRecipes")
load_poedat_as_table(dbcon, "BlightCraftingItems")
load_poedat_as_table(dbcon, "BlightCraftingResults")
load_poedat_as_table(dbcon, "PassiveSkills")

passive_annoints = dbcon.execute("""
SELECT BlightCraftingItemsKeys as oils, PassiveSkills.Name as passive
FROM BlightCraftingRecipes
JOIN BlightCraftingResults ON BlightCraftingResults.Row = BlightCraftingResultsKey
JOIN PassiveSkills ON PassiveSkills.Row = PassiveSkillsKey
""").fetchdf()
#

I got what I want ^^

#

I'll make an issue though

velvet fog
#

huge changed on GrantedEffect

worthy cape
worthy cape
#

Completely new files:

Data/GrantedEffectStatSets.dat64
Data/GrantedEffectStatSetsPerLevel.dat64
worthy cape
#

Seems like they're kind of doing the same thing as old GrantedEffects and GrantedEffectsPerLevel but with arrays instead of Stat1..6.

#

Time to sleep, but the schema is probably somewhat straightforward if one squints at the old files.

#

Leaving off with this guesswork:

type GrantedEffectStatSets {
  Id: string
  _: [rid]
  StatsKeys: [rid]
  StatsValues: [i32]
  _: f32
  _: f32
}
type GrantedEffectStatSetsPerLevel {
  _: rid
  GemLevel: i32
  PlayerLevel: i32
  _: i32
  _: i32
  _: i32
  _: i32
  _: [rid]
  _: [rid]
  _: [rid]
  _: [rid]
  StatInterpolations: [i32]
  _: [f32]
  _: [i32]
  _: [i32]
  _: [rid]
}
odd sail
#

for the two f32 in GrantedEffectsStatSets it's BaseEffectiveness and IncrementalEffectiveness

# 
  _: rid

That first rid maps to GrantedEffectsKey

#

I would rename PlayerLevel: i32 to PlayerLevelReq: i32

#

After PlayerLevelReq it's CritChance (as an Int * 100)

worthy cape
#

Yeah, pulled names out of my behind, I'm sure there's historical names and established patterns.

odd sail
#

All the

  _: [rid]
  _: [rid]
  _: [rid]
  _: [rid]

Map to Stats

#

some of them are level, level which map to resolved damage values

odd sail
#

@worthy cape

type GrantedEffectStatSetsPerLevel {
  GrantedEffectsKey: rid
  GemLevel: i32
  PlayerLevelReq: i32
  CritChance: i32
  AttackTime: i32
  BaseMultiplier: i32
  DamageEffectiveness: i32
  AdditionalFlagsStatsKeys: [rid]
  FloatStatsKeys: [rid]
  InterpolationBasesKeys: [rid]
  AdditionalStatsKeys: [rid]
  StatInterpolations: [i32]
  FloatStatsValues: [f32]
  BaseResolvedValues: [i32]
  AdditionalStatsValues: [i32]
  _: [rid]
}```
velvet fog
#

GrantedEffectsKey -> GrantedEffectStatSetsKey
_: [rid] -> GrantedEffectsKeys: [rid]

velvet fog
#

got issues on InterpolationBasesKeys and StatInterpolations, it can't cover all Stats now.

#

sizeof(StatInterpolations)=sizeof(FloatStatsKeys)+sizeof(AdditionalStatsKeys)
sizeof(InterpolationBasesKeys)=sizeof(FloatStatsKeys)

odd sail
#

there seem to still be some Boolean (or what I'm calling now Flags) Stats in the GrantedEffectsPerLevel DAT file

#

@velvet fog did you by any chance update the structure of GrantedEffects and GrantedEffectsPerLevel?

velvet fog
#

GrantedEffects removed BaseEffectiveness and IncrementalEffectiveness, others are the same

odd sail
#

yeah, it's GrantedEffectsPerLevel that's a mess mostly

velvet fog
# 
type GrantedEffectsPerLevel {
  GrantedEffectsKey: GrantedEffects
  Level: i32
  LevelRequirement: i32
  CostMultiplier: i32
  StoredUses: i32
  Cooldown: i32
  CooldownBypassType: CooldownBypassTypes
  VaalSouls: i32
  VaalStoredUses: i32
  CooldownGroup: i32
  _: i32
  VaalSoulGainPreventionTime: i32
  AttackSpeedMultiplier: i32
  _: i32
  CostAmounts: [i32]
  CostTypesKeys: [CostTypes]
  ManaReservationFlat: i32
  ManaReservationPercent: i32
  LifeReservationFlat: i32
  LifeReservationPercent: i32
  AttackTime: i32
}

@odd sail

odd sail
#

so there is some redundancy it seems

#

LevelRequirement and AttackTime seems to exist in both GrantedEffectsPerLevel and GrantedEffectStatSetsPerLevel

velvet fog
#

GrantedEffectStatSetsPerLevel.AttackTime, it's AttackCritChance in dat-schema

odd sail
#

you sure about the CooldownBypassTypes?

velvet fog
#

not sure

odd sail
#

it uses 4 for Fireball and the CooldownBypassTypes only has 3 rows

velvet fog
#

oh, confirmed

#

I though it's a bug on ggg's side

#

because all [1, 2] values are correct matched old values

odd sail
#

I'm getting a different order for variables then you have above for GrantedEffectsPerLevel. As an example, CostAmounts: [i32] seem to be 12th variable, not 15th

odd sail
#

using Fireball as an example

velvet fog
# 
{"ID":"0",
"GrantedEffectsKey":"0",
"Level":"1",
"LevelRequirement":"1",
"CostMultiplier":"100",
"StoredUses":"0",
"Cooldown":"0",
"CooldownBypassType":"4",
"VaalSouls":"0",
"VaalStoredUses":"0",
"CooldownGroup":"-1",
"Val10":"-30",
"VaalSoulGainPreventionTime":"0",
"AttackSpeedMultiplier":"0",
"Val13":"0",
"CostAmounts":"6",
"CostTypesKeys":"0",
"ManaReservationFlat":"0",
"ManaReservationPercent":"0",
"LifeReservationFlat":"0",
"LifeReservationPercent":"0",
"AttackTime":"0"
}
odd sail
#

I see... I assumed CooldownBypassType was a rid

velvet fog
#

nope :p

odd sail
#

to the CooldownBypassTypes.DAT

velvet fog
#

it's a 12 bytes empty dat, so only use 4 bytes when rid

odd sail
#

@velvet fog regarding not mapping all stats. Looks like they moved Constant Stats to exist in GrantedEffectStatSets while per level stats exist in GrantedEffectStatSetsPerLevel

velvet fog
#

ya

odd sail
#

Using Fireball as an example again - base_chance_to_ignite_% is in GrantedEffectStatSets while all the other stats are in the Per Level one

velvet fog
# 
type GrantedEffectStatSets {
  Id: string
  ImplicitStatsKey: [Stats]
  StatsKeys: [Stats]
  StatsValues: [i32]
  BaseEffectiveness: f32
  IncrementalEffectiveness: f32
}
odd sail
#

ya... I just called StatsKeys differently ... ConstantStatsKeys

#

but format is same

velvet fog
odd sail
#

that's b/c you are an over-achiever 😛

#

@velvet fog you might want to change that to # to # instead

Blah.png
velvet fog
velvet fog
velvet fog
simple ravine
#

Wonder if PoE 2 will have higher resolution models

worthy cape
#

Bah, I was going to show a similar screenshot but my project apparently is still on 3.16 ^_^

#

Guess it's a good motivator to get the next generation of file collective online.

slate rover
#

does anyone know if support gem letter is somewhere to be found in de data? Since its removed from GrantedEffects with 3.18.1

hazy fiber
#

havnt looked but I would assume with the rest of the constants?

slate rover
#

which constants do you mean? SkillGems.dat?

carmine merlin
slate rover
#

oh i see, the spec was wrong. thanks

worthy cape
#

3.18.1 had some turbulence around GrantedEffects and friends, see recent discussion here.

carmine merlin
#

spec changed with the update yesterday

slate rover
carmine merlin
slate rover
#

thanks, will take a look at it later today

velvet fog
#

https://poedb.tw/us/model
Model Viewer for monsters and uniques

Here is a model viewer that can display about 2,100 monsters and unique item models. Still trying to convert more models to .obj format.

Credit to zao at https://gitlab.com/zao/poe-cpp/-/tree/squint-prototype/libpoe/poe/format.

Screenshot: https://i.imgur.com/S2Hs89n.png

more discussion in official discord #tooldev-general

poedb
PoEDB, Path of Exile Wiki

PoEDB provides things come out each league, as well as items, uniques, skills and passives. Path of Exile Wiki editing functions.

GitLab
Files · squint-prototype · Lars Viklund / poe-cpp · GitLab

GitLab.com

Imgur
worthy cape
#

I'm pleasantly surprised by how fast you got it up to speed again, nice job.

#

I've got to pick up the pace on my stuff again, finally got some download automation going with dedup for LAN serving.

timid grotto
#

Isn't that model project a violation of ToS 7a,b&i?

velvet fog
#

@worthy cape VtxNormal_4i8, I don't understand how it convert to

vn i j k

    Polygonal and free-form geometry statement.

    Specifies a normal vector with components i, j, and k.

    Vertex normals affect the smooth-shading and rendering of geometry.
    For polygons, vertex normals are used in place of the actual facet
    normals.  For surfaces, vertex normals are interpolated over the
    entire surface and replace the actual analytic surface normal.

    When vertex normals are present, they supersede smoothing groups.

    i j k are the i, j, and k coordinates for the vertex normal. They
    are floating point numbers.
worthy cape
#

In modern formats (mLOD) the normals are encoded in 8-bit SNORM: https://docs.microsoft.com/en-us/windows/win32/direct3d10/d3d10-graphics-programming-guide-resources-data-conversion#data-type-terminology and have IJK in the first three bytes.
In legacy formats, the normals are similar, but offset by +128 in the stored data. You can convert them to the future format by subtracting 128 like in this code: https://gitlab.com/zao/poe-cpp/-/blob/squint-prototype/libpoe/poe/format/dolm.cpp#L171-198
Once you have a consistent SNORM representation for them, they can be converted to the [-1.0, 1.0] range that OBJ expects by following the logic in the DirectX documentation table.
You should be able to emit vn lines in much similar ways to how you emit v and vt lines, later referencing them from f v/t/n entries.

#

The same convention holds for the tangent but you can't express that in OBJ and as such, cannot do normal mapping with textures. The fourth byte of normals I don't believe is used for anything in materials. The fourth byte of tangents indicate the parity of the tangent frame, that is, if you need to flip the bitangent when computing the TBN matrix.

#

If you wish to interpret the meshes and materials in a bit more modern way on the web, you may have to look into emitting glTF (.glb) files rather than .obj.

#

You would still not get the sometimes quite elaborate shader manipulation that materials do to some meshes and surface properties, but you would at least get a shot at using the texture maps in a PBR way, and also the ability to do animations eventually.

#

That of course requires that I or someone finish up the work on animation (.ast) files and write those up. My last parser for those were version 6, while I've seen at least version 8 in the wild now.

velvet fog
#

I haven't been touched to glTF before, time to study it

worthy cape
# timid grotto Isn't that model project a violation of ToS 7a,b&i?

I'm not gonna lie, much of tooldev is flirting with section 7 quite a bit indeed. You kind of have to look at the purpose of the work and how much of a positive impact it's making on the greater game and community.
Resources like the wiki and PoB are pretty much universally powers for good, and as is PoEDB most of the time even if there's a bunch of early league content and mechanisms that they rather would like people to discover organically.

Personally I've put my limit at not reverse-engineering actual code from the executable to determine things about file formats and how they're processed, but in my world the shape and meaning of the data formats themselves are an open to be solved and to build responsible tools around. I do recognize that by documenting them it may make it easier for bad actors to have more context and capability to do naughty things but on the whole it feels like a positive thing which again is at the core of our tooldev community.

There's a lot of things I'm still reluctant to carry to completion like my whole data delivery system which can provide arbitrary access to any Path of Exile file since version 1.0.0 through the internet without having the game installed, as that's kind of very blatantly distributing a whole lot more types of files indiscriminately with way too little control over who consumes it. I only made that as a potential way forward for myself and chosen tooldevs as I was explicitly denied by support to read from the user's GGPK file in my free-standing hideout editor tool.

Thank you for coming to my TED talk.

worthy cape
#

I don't even have a remote repository for that codebase, that's how experimental it was 😄

#

It's pretty much that you dump binary blobs of indices and vertex components into a memory block, make accessors for each vertex component, making views into the block. Never got around specifying materials.

golden bane
#

I get why you don't want to RE the client, but even that is in murky waters in some cases. Without it, we would have probably not been able to figure out the bundle hashing for example

worthy cape
#

The only client RE I've done has been to see if I can get a lead on the possible cause of bugs like the AVX2 use on macOS, the accessibility API misuse that got sticky keys enabled by accident, macOS rendering being clamped at 60Hz on faster monitors, and looking for version number strings for my data ingestion. Most of that again for the good of the community or personal curiosity.

worthy cape
golden bane
#

There is a reason why RE is legal under specific circumstances in some countries (e.g. all EU members) to provide compatibility or be able to fix critical bugs yourself

velvet fog
violet path
#

my shredded cheese trying to create 3d models of what eats it in the middle of the night

slate rover
#

Does anyone know if there is a mapping for the stat transformations for Lioneye's Fall? Like how to go from two_handed_melee_physical_damage_+% to physical_bow_damage_+%

velvet fog
#

As far as I know, no

slate rover
#

cant image they hardcoded those transformations

velvet fog
#

There are many server side .dat files

slate rover
#

seems like something that the client would need for rendering the passive tree

#

but i guess server could send that once

worthy cape
#

Kind of like a timeless jewel mangles nodes?

slate rover
#

hmm yeah i suppose

#

but lioneyes doesnt seem like something you need to hide, unlike timeless jewels

velvet fog
#

At least 40 server-side .dat files with binary symbols but not in .ggpk

slate rover
#

guess ill create my own mapping

visual flame
#

Good afternoon!
Can someone provide some insights on how to attain an API key for PoE.
I already have a PoE account read the docs and made no sense on how to create one!
Any insights will be much appreciated!

compact isle
visual flame
worthy cape
#

For proper API access the immediate step is OAuth with all that entails. You set that up by contacting the oauth@ email address with the details they desire. This means that your project will need to have a secure server that runs the whole OAuth dispatching stuff for your clients as you can't have any of that running client-side, clients need to do the auth procedure to bounce via the GGG OAuth sign-in system to grant your application rights.
If you're looking to do more client-originating stuff or toy around, some of the API and the website still accepts anonymous requests or your personal POESESSID to access your own information.

#

You need to figure out what scope (parts of the API) that your application needs access to and then talk with GGG for the setup details on that.

#

The user-facing flow is similar to something like what you see on filterblade where you've got a "Sign in with PoE" like interface.

#

What kind of application are you intending to make and does it fit into this square hole? 😄

#

I'm not super well versed in this side of PoE tooldev but I'm sure there's some peeps here that are more aware of things.

visual flame
#

It was an addition an existing discord bot, was hoping to tie my account to my discord bot and utilize trade-search functionality auto tasks driven by command/messages.

#

I've seen the PESESSID access looked appealing was trying a proper access!
Thank you that makes more sense, I went back to the doc, after Novynn's post and read what you mentioned as well that I need to contact them via e-mail.

#

This makes sense why it was so confusing so far#

worthy cape
#

The trade site is quite disconnected from the "traditional" API that this site outlines.

#

The interface that people access the trade searches and stuff with is the same as an interactive browser uses to make searches, mostly observed by squinting at the Developer Console.
Auth for that is either anonymous or signed in (i.e. POESESSID cookie).

#

If you're looking for prior art on how to do it, the source for Awakened PoE Trade might be educational if you know how to read whatever Electron script it's built with.

visual flame
#

Brilliant, actually with this information alone I will be able to pull through!
Thank you zao!

worthy cape
#

Don't forget to honor the rate limit headers, be nice to the servers and remember that trade queries have a maximum complexity on how complex they can be.
Also set an user agent that identifies your software, its version, and contact information to you.

#

Makes it easier to block a version or reach you if it ends up doing something bad.

visual flame
#

Absolutely! Thanks alot!

worthy cape
worthy cape
#

Heh, I'm quite behind on my data ingestion... found out now that with 3.17 a lot of dialogue audio files were moved from being loose to bundled.

#

Also found that I've got like 1.7 GiB of duplicate audio files ^_^

rapid pagoda
worthy cape
#

"huh, odd, this request to my API seems stuck"

#

(plz2nothack)

buoyant cobalt
#

i made this xD

unknown.png
frank drift
#

counting is hard apparently

#

j/k

buoyant cobalt
#

i made cause i was so annoyed when i had to move cursor there im so lazy

oak estuary
#

Dang, I really liked that one

golden bane
worthy cape
#

Two of my internet-facing "servers" are old laptops, this one was decommissioned from work because it was "unstable".

#

USB3 xHCI has a tendency to fall out of existence sometimes too, so might've gotten confused around that, but CPU was a bit toasty too.

#

I've been a bit tempted to go Hetzner for this project, but that means shelling out like 40 EUR a month for a capable enough dedi.

golden bane
#

But shouldn't kernel code be unable to stall? Iirc it's written in a way that loops always return eventually

golden bane
worthy cape
#

Physical console was responding enough to give me a login prompt when I hit Enter, but nothing more.

golden bane
#

When you ssh into the quantum compute clusters, it usually takes minutes at a time to do anything

#

But they always respond, eventually

simple ravine
#

Hehe, bringing in quantum computer clusters to solve PoE problems, that'd be something 😇

frank drift
#

pretty much just confirming what was already known

carmine merlin
#

Just need someone to wrongly explain how the Timeless jewel algorithm works and Mark will reveal its secrets 🙏

frank drift
#

oldest trick in the book

rapid pagoda
long sky
#

Looks like cloudflare just had an outage in case anyone was wondering why a bunch of stuff was down

frank drift
#

yup

#

look forward to seeing what happened

tawny agate
#

It's always DNS

worthy cape
#

I've finally gotten close to the finish line for my asset server system to the point where I can try parsing newer game files than late 3.16. This is going to be fun.

worthy cape
#

Already ran into some new format versions: v6 SM, v7 TDT, mLOD vertex formats 58/62, FFX files seem to have changed a bit, quite a bit of AO/AOC parsing mishaps but those formats are cursed.

# 
Format .ao: 1 bad, 44744/44745 good
Format .aoc: 61 bad, 44729/44790 good
Format .arm: 0 bad, 21578/21578 good
Format .dgr: 1 bad, 2270/2271 good
Format .et: 0 bad, 1339/1339 good
Format .ffx: 1 bad, 24/25 good
Format .fmt: 2 bad, 25525/25527 good
Format .gft: 0 bad, 853/853 good
Format .gt: 0 bad, 1017/1017 good
Format .rs: 0 bad, 909/909 good
Format .sm: 0 bad, 26759/26759 good
Format .smd: 2 bad, 26048/26050 good
Format .tdt: 51 bad, 14370/14421 good
Format .tgm: 0 bad, 80767/80767 good
Format .tgr: 0 bad, 701/701 good
Format .tgt: 0 bad, 18687/18687 good
Format .tsi: 0 bad, 1002/1002 good
Format .tst: 0 bad, 868/868 good
#

Less fallout than I expected, the above is from 3.18.1.

#

I ran into the dumbest problem last night when populating my local cache of files... I rapidly ran out of TCP ports on my desktop machine 😄

#

The delivery infrastructure uses HTTP(S) and fetches each one individually... all 780k of them. As the library I'm using doesn't do any pipelining or socket reuse, the machine almost immediately stops having any ephemeral ports available as they're all in TIME_WAIT state.

#

I should move this to instead using my own remote bundles, kind of like poe-dat-viewer but with a private source.

graceful barn
#

having trouble tracking down the source of a buff in PoB, on my dev version I was noticing a difference in Physical damage reduction from my version to the regular PoB release, the highlighted buff doesn't appear to have an aura as its origin, and I'm not sure how to trace down what's causing it

unknown.png
#

every other aura follows the pattern of aura, damage, speed, PDR, but that one seems out of place

#

on my dev version, those 3 lines don't seem to exist

unknown.png
#

if its useful for replicating, https://pastebin.com/DDYL6ttL this is the pastebin I'm working from, and that box is from hovering "Aura and Buff Skills' under the minion stats on the calc panel

Pastebin
Untitled - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

worthy cape
#

That could be fun to hunt down, considering that dev has seen a flurry of work on getting the 3.18.1 GrantedEffects clusterhug under control.

#

@oak estuary Got any clue on the above?

graceful barn
# worthy cape That could be fun to hunt down, considering that dev has seen a flurry of work o...

if it helps at all, my dev version has an old version of CalcsTab.lua and CalcBreakdownControl.lua https://github.com/PathOfBuildingCommunity/PathOfBuilding/pull/2927/commits

GitHub
Rough Aurabot Exporter by NathanielHinshaw · Pull Request #2927 · P...

Super rough aurabot exporter, requires you to add a minion to the build, hover over the calc'd buff window on the calc's tab with minion stats selected, and then click the export button. f...

worthy cape
#

Oh, even worse 😄

graceful barn
#

just trying to figure out which end is wrong

#

wouldn't have noticed it, but I was validating my Physical Damage reduction calculations with the release PoB version, and couldn't seem to find why my code was off by 4%

#

functionally those are the only two files changed in my PR though

#

Normally I would assume the old code was at fault, but it’s strange that it breaks from the pattern of aura -> buffs

worthy cape
#

@carmine merlin You've been poking around this stuff lately, right?

graceful barn
#

for the sake of further information, I've narrowed it down to not being in CalcBreakdownControl, as I updated my version of that and still see a value that doesn't match the current release

carmine merlin
#

I'll squint at the spaghetti and see what it tells me

graceful barn
#

by comparison, this is on the official 2.18.1 release

unknown.png
#

so perhaps the issues already been fixed? I'm not sure how the release pipeline looks on that

carmine merlin
#

So was the issue that you now have more pdr than before and you're not sure why?

graceful barn
#

after vaal clarity

#

there are 6 buffs before before the next aura

#

when I only expect 3

#

that being, damage -> speed -> pdr

#

on the official 2.18.1, I see damage -> speed -> pdr, and then another set of damage -> speed -> pdr before vaal grace

#

I was adding PdR into my aurabot exporter, and was checking against the release PoB version to make sure I was calculating it correctly, but I was 4% off

#

as a result of that second set of buffs

carmine merlin
#

My dev version doesnt have the issue the live 2.18.1 has for some reason

#

Seems it was fixed/changed when we altered the GGPK exporter after GGG changed the data files in the 3.18.1 patch

graceful barn
#

certainly an interesting bug

#

for the sake of it, I updated my CalcsTab and CalcBreakdownControl on my PR to align with any changes made, and see the same behavior as I saw initially, I'll assume for now that this is accurate

#

that being the same as the behavior the dev branch currently has

#

Thanks for the help!

carmine merlin
#

Tracked it down to being an issue with Vaal Discipline being on

#

For some reason on the dev branch, enabling it doesn't give you pdr. Can you see if that's the same issue you were having @graceful barn

graceful barn
carmine merlin
#

It should in game though right?

graceful barn
#

as well as no effect on attack speed

#

or damage

#

yeah

#

every aura should individually apply one of each of those buffs

#

multiplied by effect

carmine merlin
#

Hmm, so current master is correct and current dev is wrong

graceful barn
#

Yes, that does appear to be the case

carmine merlin
#

Definitely a Vaal Discipline issue. Empty build with just that gem has the same issue

graceful barn
#

for reference, the 3 mods would be "Auras from your Skills grant 2% increased Damage to you and Allies", "Auras from your Skills grant +1% Physical Damage Reduction to you and Allies", and "Auras from your Skills grant 2% increased Attack and Cast Speed to you and Allies"

#

my Lua is pretty sketchy, coming from an embedded C background, so I'm not sure I'll be much help tracing it down, even my PR was just a hacky way to export the data I wanted

#

but I can confirm that each of those mods should multiply off Vaal discipline

#

as well as against every other aura

worthy cape
#

@velvet fog The two new vertex formats seem related.

Vertex format 62 (from SMD) is 36 bytes and something like { f32_3 position; u8_4 normal; u8_4 tangent; f16_2 texcoord; u8_4 bone_indices; u8_4 bone_weights; u8_4 unk; }.
The bone assignments are a bit uncertain as I have a limited number of specimen but they're like [01 00 00 00; FF 00 00 00].
The final four values are like [FF 00 00 FF] so my guess is that it's possibly a four-component RGBA8-like colour value.

Vertex format 58 (from FMT) is 28 bytes and something like: { f32_3 position; u8_4 normal; u8_4 tangent; f16_2 texcoord; u8_4 unk; }.
The final four values are something like [00 00 00 FF] so again possibly a vertex colour.

worthy cape
#

Probably not of interest to anyone, but TDT v7 has a new stringref before the common TGT stringref in the header and a new stringref + byte at the end of the file after the TMD stringref.

#

Purposes unknown as they're all empty strings in the few files I've manually parsed thus far.

#

Oh boy, InputOutput.ffx is serving up a neat little curveball with a whole new EXTENSIONPOINT concept.
Since 3.17 somewhere between 3.17.0 and 3.17.4c:

EXTENSIONPOINT VertexTangent
    group Material
    vertex
    defaultstage WorldTransform
    type float4
    inout VertexData vertexData : vertex_data
    write: {{ vertexData.tangent = value; }}
    read: {{ value = vertexData.tangent; }}

In 3.17.0:

FRAGMENT VertexTangent
    cost: free
    vertex
    write
    matches InputVertexTangent
    defaultstage WorldTransform
    in float4 input
    inout VertexData vertexData : vertex_data
{{
    vertexData.tangent = input;
}}

FRAGMENT InputVertexTangent
    cost: free
    vertex
    read
    matches VertexTangent
    defaultstage WorldTransform
    inout VertexData vertexData : vertex_data
    out float4 output
{{
    output = vertexData.tangent;
}}
#

The parser for FFX is gnarly already, this isn't going to help 😄

oak estuary
#

New phone didn't give me Discord notifications :grumble: @carmine merlin it's because the new Vaal Discipline doesn't have base_maximum_energy_shield as a stat anymore, so it's not getting the aura from the statMap

frank drift
#

BGP strikes again

worthy cape
#

Ah, SM v6 is a bit enlightening. There's now BoneGroups 0 directives.

worthy cape
#

Not quite as enlightening as I thought, the value is never anything but 0. I assume it's a count of subsequent lines like this, but without specimens I can't tell:

Materials 3
    "Art/Microtransactions/Aura/maligaro_restraint/gold/dark_goldc.mat" 1
    "Art/Microtransactions/Aura/maligaro_restraint/gold/goldishc.mat" 1
    "Art/Microtransactions/Aura/maligaro_restraint/reflectionc.mat" 1
glad ravine
#

hi I want to translate the RePoE mods and have a problem:
from stats.json:
{'id': 'local_physical_damage_+%', 'max': 154, 'min': 135}
currently matches to the first element from stat_translations.json:
[
{'condition': [{}, {'min': 1}], 'format': ['ignore', 'ignore'], 'index_handlers': [[], []], 'string': 'No Physical Damage'},
{'condition': [{}, {'max': -1}], 'format': ['ignore', 'ignore'], 'index_handlers': [[], []], 'string': 'No Physical Damage'},
{'condition': [{'max': -100}, {}], 'format': ['ignore', 'ignore'], 'index_handlers': [[], []], 'string': 'No Physical Damage'},
{'condition': [{'min': 1}, {'max': 0, 'min': 0}], 'format': ['#', 'ignore'], 'index_handlers': [[], []], 'string': '{0}% increased Physical Damage'},
{'condition': [{'max': -1, 'min': -99}, {'max': 0, 'min': 0}], 'format': ['#', 'ignore'], 'index_handlers': [['negate'], []], 'string': '{0}% reduced Physical Damage'}
]
because {} in {'condition': [{}, {'min': 1}] accepts any min and max value
can someone help me understand it?
(the documentation did not help with this problem https://github.com/brather1ng/RePoE/blob/master/RePoE/docs/stat_translations.md)

craggy iron
#

I'm having problems with my PoB fork
Here's a description of what Ive got to deal with(s/s's won't really show anything, and I dunno how to makes a video of whats going on, on my desktop screen)

Once I open PoB Fork, everything opens up perfectly fine, but once I try clicking on something nothing happens. But if I mix/max the screen anything I've clicked on appears. Below is the best description of the problem I'm having with PoB Fork.

( I can do multiple clicks before min/maxing the screen, but since some selections cause the next selection box to move down slightly I try to select 1-2 things before I min/max he screen to have what I've selected show up.)

Click import build.
Mix/Max screen for import list to come up.
Select/Click my character from drop down list.
Min/Max screen.
Click the 3 boxes to not remove passive tree, and items. Then click import.
Again... Min/Max screen. ETC

Also hovering over any skill tree nodes or anything where a popup mini-screen/bubble that displays whatever your mouse is hovering over does not show either.

I'm sure you get the jist of what I'm trying to explain here now. I've also tried uninstalling, and reinstalling PoBFork, but it still gives me the exact same problem each time. I thought maybe it had something to do with the settings in the SimpleGraphic.cfg, but I'd rather not make any changes which could lead to making it worse.

Any help regarding this would be much appreciated!

hazy fiber
#

sounds like PoB isnt updating its render between frames, imo it probs thinks its in the background, but not actually sure, sounds somewhat similar to the blackscreen bugs

#

@worthy cape thoughts? ^

#

you said last time thats probs not whats happening

worthy cape
#

Sounds quite similar.

worthy cape
#

I wonder if we should make a cvar knob to turn off the rate limiting and some of the focus logic.

worthy cape
#

Heh, this is neat. I've bisected my way down to the patch that introduced AST v11, namely 3.13.2 172073.
The patch notes for that one says:

Various renderer systems have been made roughly 15-20% faster, which doesn't necessarily translate to a performance improvement for all users but anecdotally it seems like it's around 5% faster on average for us.

#

Before that the highest AST was v9, and there doesn't seem to ever have been a public release with a v10 AST so I can't nail that format down.

#

I wonder if it coincides with mLOD adoption or some other big format revision.

waxen musk
#

hey what happened to GrantedEffectsPerLevel, none of the tools can open it, did they change rowsize recently?

#

specification size 245 vs real size 96

long sky
waxen musk
#

uh what version is the game on now?

worthy cape
#

In summary, it's been largely shuffled off into StatSets files and has had a bit of rework around which stats and values are represented. Chuan and PoB are the resident experts on it as you can see from cyborg's linked discussion.

#

3.18.1.2

waxen musk
#

thats different from 3.18 i guess?

worthy cape
#

This arrived with 3.18.1

waxen musk
#

wish i didnt update my game 😐

worthy cape
#

You can download old versions of the game with SteamRE/DepotDownloader if you wish to have a known good dataset to work with.

worthy cape
#

You want -app 238960 -depot 238961 -manifest 8628579843003481755 to grab 3.18.0e 196649.

worthy cape
#

Now that was a fun evening of debugging. Ran into a Mermaid packet that ooz couldn't decompress nor could any actual Oodle DLL between v3 and v5 for reference.
Found the root cause and am extremely uncertain of the repercussions of modifying the decompressor 😄

#

I don't think that any of you have run into this problem in the wild, as I triggered it after finding a nested bundle in a data file I doubt anyone even knew contained nested bundles.

rapid pagoda
#

Wait, nested bundles are a thing?

worthy cape
#

Well, not bundles per se as they don't contain files.

#

Compressed data with the same header structure as a .bundle.bin file.

#

This looks like a good stopping point for tonight:

Unhandled exception at 0x00007FFFFA8BE129 (libooz.dll) in sweep.exe: Stack cookie instrumentation code detected a stack-based buffer overrun.
rapid pagoda
#

Mmm, cookies.

#

Curious how/where you're running into this

worthy cape
#

AST (skeletal animation data) files v8 through v11 (highest known) have the actual animation tracks in a compressed payload at the end of the file.

#

Earlier versions of AST had it uncompressed in-place.

rapid pagoda
#

Ahh, OK. Interesting

worthy cape
#

I've mapped out the bones and their relationships, the animation names and their byte ranges in the payload, shapes (point lights, I think), and finding that it was indeed compressed data at the end.

#

I've got some old work on v6 AST files where I had parsed out individual animation tracks for the keyframed bones but haven't related that to the new storage.

#

Went on a multi-day goose chase where I tried to make sense of the compressed data without noticing the pair of bytes that screamed "Kraken" at me in the hex editor, thinking it was just some clever scheme with denser and smaller data types.

rapid pagoda
#

Weird that they'd be compressing data inside an AST when that's already compressed in a bundle -- unless they're using some kind of super special purpose compressor

worthy cape
#

I have a bit of a limited sample set currently thanks to my new data delivery service only supporting bundled releases, but it's quite possible that the AST use predates bundles.

odd sail
#

@velvet fog hey, you around?

velvet fog
#

ya

odd sail
#

any idea how you get a map to be above Tier 17?

#

that's using the level 84 monster life of 16265 * 1.4 monster variety multiplier * 2.17 map multiplier = 49413

#

However, the 2.17 map multiplier can be higher for higher tiers

x 2.17 (level 84 - Tier 17 maps)
x 2.25 (level 85 - Tier 18 maps)
x 2.34 (level 86 - Tier 19 maps)
x 2.42 (level 87 - Tier 20 maps)
x 2.50 (level 88 - Tier 21 maps)
x 2.58 (level 89 - Tier 22 maps)
x 2.66 (level 90 - Tier 23 maps)
#

I just don't know how you get higher tiers... so curious

velvet fog
#

The pagination of lvl is just a few levels I picked at random, because showing all the lvl would be too messy

odd sail
#

I understand, but been having a 2-3 day debate with some Unearth -> Detonate Dead users about how to calculate the corpse life

#

so a build can get to spawning a Level 95 Bone Archer (via Unearth) which leads to 31220 (base) * 1.4 (variety multiplier) * 2.17 (map multiplier for level 84 areas) = 94846

#

you used to be able to get higher Map Tier levels with Watchstones (Ivory) previously, but now we only have 4 voidstones... just curious if there is a way to hit these higher multipliers

velvet fog
#

no ideas

frank drift
#

is it purely area level, or monster level? there's two atlas tree notables that increase monster level, but none that increase area level

#

aside from the pinnacle boss arena area level ones of course

odd sail
#

Monster Level might raise the Level 95 Bone Archer to Level 96 or 97, but not the map multiplier I believe

frank drift
#

you would get the "x 2.25 (level 85 - Tier 18 maps)" with the pinnacle boss area notables, and I imagine that build is primarily concerned with doing things like uber bosses anyway

hazy fiber
#

and if monster level worked on them it would push it to 100, but iirc no monster level modifier applies to the monster directly

#

either it increases the level of the area, which increases the base level

#

or it increases the level of the loot dropped, and not the monster itself

#

or in the case of strongbox, that does increase the monster level, but they cant spawn unearth

worthy cape
#

Ugh, I really don't know where to go now with ooz as I don't understand the bitstream and format remotely enough to make competent fixes.

worthy cape
#

Ho ho, fixed the stack smash.

#

Format .ast: 0 bad, 15022/15022 good

#

The stack smash in Leviathan I can fix trivially but I'm not sure about my fix for Mermaid as that's bumping the scratch size by an arbitrary amount of 2k without knowing if this will eat too much scratch or not be sufficient.

worthy cape
#

Went conservative with the Mermaid Tans Luts, opting to use up to 16 KiB as that's the maximum size of Lut space that the bitstream can indicate but it's still a bit iffy as I don't know if this will slam into the scratch headroom somewhere.

#

It should be safe at this point as we've got the full 432 KiB of scratch space available there.

worthy cape
#

@velvet fog This is nice, v8 of AST seems to have the same kind of bone tracks as v6 did.

unknown.png
#

At least based on the two files I've manually run my template on.

rapid pagoda
worthy cape
#

Unfortunately, GIGO.

worthy cape
radiant ermine
#

hey guys, im getting an invalid token error when using it in an API call, even after getting the right token from "https://www.pathofexile.com/oauth/token"

# 
{'access_token': mytokenishere, 'expires_in': None, 'token_type': 'bearer', 'scope': 'account:stashes'}

this is the response i get, and i passing my token in as a header into the get stashes api call like this

    header = {
        "User-Agent": f"OAuth {client_ID}/{version} (contact: {contact})",
        "Authorization" : "Bearer {token}"
    }
#

anyone got a clue?

# 
{"error":"invalid_token","error_description":"The access token provided is invalid"}

this is the error im getting

worthy cape
worthy cape
#

I've blocked out the animation tracks but haven't looked at generating the animated poses from that.

worthy cape
worthy cape
#

There's a whole bunch of stuff I don't understand yet, most of the portals don't do much at all, probably something with the secondary channels that sometimes occur in animation tracks, and there's a whole bunch of flags that I ignore 😄

bright ridge
#

Copying and pasting my message from TFT dev chat:

Anybody know where I can contract an artist from the PoE community? Looking for some new icons for Chaos Recipe Enhancer, relatively simple stuff

Else I'll just hit up Fiverr or something

#

Also still interested in getting my dev role if anyone can point me in the right direction for that, I'm the dev for the Chaos Recipe Enhancer (aka Chaos Recipe Tool)

hazy fiber
#

ferrets gone, one sec

#

@bright ridge this ^

#

which uh, maybe not so coincidentally, was also you asking about it

bright ridge
#

Haha sounds good. Messages have been sent and the ball is rolling.

On the original request, I'm surprised there isn't any sort of #fan-art sorta channel on this main PoE discord

carmine merlin
bright ridge
#

Good idea! Thanks. I'll check it out

waxen musk
#

does anyone have a link to recent dat file changes?

worthy cape
waxen musk
#

ok ill check, trying to get RePoE to work with new data

#

uh how does that work? the version files are only for league mechanics data?

worthy cape
#

The files should be considered collectively, the versioned names contain tables that pertain to that league in particular.

#

Like here you have the Drone and Sentinel type tables in 3_18_Sentinel while the StatSets ones are in _Core as they're not particular to the league.

#

For PyPoE in particular, I think Brathering had some logic to generate specs for their version of PyPoE at some point in time, not sure if that's still being tried.

waxen musk
#

hmm not sure where to find that 😐

#

where does it get the specs from?

#

it does seem they're generated from something

worthy cape
waxen musk
worthy cape
#

Historically PyPoE's stable.py has been manually maintained, there's been recent work across the community to gather around the dat-schema repo for a more collaborative and central schema, used by among others poe-dat-viewer.

waxen musk
#

i guess the script updated it

worthy cape
#

The script I linked above seems to synthesise stable.py from the GQL schema, but I don't know if it's the "proper" way to do it.

waxen musk
#

welp now repoe doesnt work because it tries to get old fields 😄

worthy cape
#

Haven't touched that script much apart from trying it in the wiki's fork of PyPoE and breaking all the things 😄

#

One of the core problems with replacing the old manual schema with a generated one is that there's historical choices of field names that probably don't agree anymore.

waxen musk
#

oh it has manually edited virtual fields which dont work with new spec it seems

#

ah indeed, CostTypesKeys got renamed to CostTypes for some reason >.>

civic crane
#

its a redundant info, you can append "Keys" to field name if it's an array of references

waxen musk
#

ye but it breaks code that relied on it, seems the whole grantedeffects changed completely though

#

did they remove stats from it?

worthy cape
#

It was quite revamped, a lot of GrantedEffects and GrantedEffectsPerLevel was hoisted out into GrantedEffectsStatSets* where things are now more proper arrays than the previous Stat1, Stat2, Stat3... fields.

#

There's also been some additional changes to which stats are in which list, ultimately making it easier to piece them together but quite an upheaval.

waxen musk
#

oh that sounds fun

worthy cape
#

Among the tools, I would reckon that PoB (and PoEDB) are most up to speed on things.

#

There's a discussion in here about all the changes and I think PoB has had a bunch of commits lately to reconcile things.

waxen musk
#

PSGFile from PyPoE isnt working either 😦

worthy cape
#

This linear blend skinning is so limited in functionality, not even Tane's glass parts work 😄

velvet fog
#

I'm working on the relationship of tags to cards

#

some card locations may be data mined in the future

simple ravine
worthy cape
simple ravine
waxen musk
#

how does the game calculate passive node coordinates? seems its no longer evenly spaced around the circle, im getting weird gaps between nodes >.>

worthy cape
#

It depends on the particular orbit used, some have hardcoded spacings and some are even.

waxen musk
#

thx just found it in POB, seems to be the same function

worthy cape
#

I'm fighting the urge to work on the VR mode, that's for sure PES_PoggersVR

simple ravine
#

I mean, you could recreate the river dance party 😄

frank drift
simple ravine
#

Too bad there's not a reliable way of automatically logging drops

#

It needs pretty good attention and discipline from the player for it to be relevant, and we all know humans == errors

worthy cape
#

Something about this animation is a bit off and I can't quite tell what:

unknown.png
worthy cape
#

The game (at least on macOS), re-opens the drop sound file every time it plays.

simple ravine
#

PoE don't mem-cache the mp3s?

#

ooh

#

interesting

worthy cape
#

I'll leave the ToS speculation to you folks 😄

simple ravine
#

hehe right, well I think it is within the light-gray zone of things

worthy cape
#

Found this out the fun way when troubleshooting why custom sounds didn't work on macOS, it looked in the wrong directory.

frank drift
#

someone could come up with some sort of computer vision AI to watch every single stream on twitch and record drops

simple ravine
#

haha @frank drift that's actually a pretty fun idea

frank drift
#

the way some people analyze esports pro player vods for eg jungle pathing

simple ravine
#

what is jungle pathing?

frank drift
#

in league of legends, the pathing of the jungle player

simple ravine
#

ah gotcha

worthy cape
#

In DotA and LoL the jungler walks around the wooded areas between the lanes. Their route is their "path".

simple ravine
#

i always trolled as akali in the jungle way back, it was fun, pentakill-jungle-akali ftw

worthy cape
#

Clear specific camps first, go visit a lane, take the next one, return to base to buy, etc.

#

Very fluid role that can have quite some impact on a game if they play poorly or well, no-one understands junglers 😄

simple ravine
#

before akali became terrible that is

worthy cape
#

When doing my last set of placements before leaving LoL I got 8/10 games as jungler.

#

Had my mates on voice to coach me through it, ended up in gold ^_^

#

(support/mid player otherwise)

#

On something different, how would you peeps visualise animation timelines?

#

I guess I should take a look at the PoE tooling presentations to get some inspiration.

#

Got some weird animations I can't quite figure out and I'm suspecting that I'm either misinterpreting the data somehow or missing out on some flags.

simple ravine
#

line graph, X, Y, Z as separate lines on a timeline is what I've seen in the tools I've used, such as Houdini

#

Not sure if IK/bones etc usually is visualized like that - nothing I've played with myself

worthy cape
#

The channels I have per bone are XYZ translation, quaternion for rotation, XYZ scale.

#

There's an optional secondary set of XYZ, XYZW, XYZ as well, I think it's the same things but overlaid but can't tell.

simple ravine
#

yeah there's usually an "angle" to them to visualize easing etc

worthy cape
#

Quaternions in particular are a bit hard to show. I could draw a little widget to illustrate the space, but not sure how much that helps.

simple ravine
#

what's W in this case? the only time I've seen W is when doing UVW mapping

worthy cape
#

Quaternions are four components, XYZ are the scalars for the IJK basis, and W is the twisty bit.

simple ravine
worthy cape
#

The fourth vector component is traditionally W, particularly in homogeneous coordinates.

simple ravine
frank drift
#

I lost my only pentakill clip when plays.tv went down, malzahar support

worthy cape
#

In GLSL you have .xyzw swizzles for coordinates, .rgba for colours and .stpq for texcoords.

rigid oxide
worthy cape
#

I've been thinking of wiring up some Python snippets to be able to set up scenes and test cases.
Also been thinking about being able to drill down to bones and see how they are affected by transformations.
Both of those are a fair bit of effort and I'm not quite sure what things I need to figure stuff out in the first place. Heaven knows I have enough stuff to work on already.

worthy cape
#

So I'm looking at adding support for .bundle.bin natively to ooz.exe. Currently doing it by trying to interpret the file passed in as a bundle header + payload, only proceeding to parse it if all the values in the fairly large header make sense as a .bundle.bin. If this fails, it parses it as the usual length-prefixed (4/8 bytes) payload that ooz expects.
Do you peeps think that such autodetection is robust enough to do automatically or should it be guarded behind a file extension or flag?
For some background, this is needed to decompress data sequences consisting of multiple 256k chunks without the Oodle DLL as ooz's regular implementation chokes on the second chunk as the decoder state isn't quite what it expects at that point in time.

#

Grew a need for this myself when slicing out the animation track data from .ast files as they're an embedded .bundle.bin-like payload at the end of the file that quite often is larger than 256k.

worthy cape
#

Better-than-nothing info on a per-bone level:

unknown.png
simple ravine
#

Are there any 3d format that natively support IK/bones, like obj, gltf or something?

#

would be cool to import into a 3d program and do some toying around

worthy cape
#

glTF does animations quite flexibly.

#

There's some animations I'm unsure if they're broken out of the box or if I'm interpreting them wrong somehow.

#

There's a byte that I've assumed is frame rate that has values like 24, 30, 60, 120, 200; but some animations are extremely undersampled and slow with those, while others require it to play out.

simple ravine
#

It is not the length of the animation?

worthy cape
#

There's a cinematic ZanaLeave at 120 FPS that definitely is around that speed, 1404 time units long with upwards of 1405 keys.

#

There's also a Kurai confront_action_02 that's 30 FPS and 200 time units long, with only like 29 keys.

#

Meanwhile Kurai confront_action_01 is 30 FPS, 160 time units long and 161 keys.

#

I guess that my interpolation may be a bit weird and that's why confrontation 02 seems so off.

worthy cape
#

Ah. confront_action_02 has doubled samples through the animation, causing the jerky motion.
There's dupes at 0 & 15, 16 & 31, 32 & 47, 48 & 63, ..., 192 & 199, with a final lone value at 200.

#

I don't see how any type of interpolation would make this better.

pseudo ocean
#

(Allows you to precalculate all Timeless Jewel results without having to Divine them all)

oak estuary
#

Did you reverse engineer this from a dataset of jewels people posted or what did you do to figure it out?

frank drift
#

pretty clear that it's not from a dataset ^^

oak estuary
#

yeah....just wanted to hear it from him 😉

frank drift
#

anyone brave enough to try it?

slate rover
pseudo ocean
#

a1 is the twister state.

slate rover
#

and what do you do with the return value? used it as an index for which stat to take?

pseudo ocean
#

no, it's a bit more complicated than that. You have to loop all the possible alternate tree versions first (Data/AlternateTreeVersions.dat) to figure out which of the jewels it is (these are the possibilites: https://i.imgur.com/93uO6jH.png) and then loop through all possible atlas passives that are in radius of the jewel socket and generate a weighting map (idk if this is a real term, I just made it up because that what it looks like to me) out of Data/AlternatePassiveSkills.dat (this files contains all the passives a timeless jewel can add/replace with, besides keystones). then roll a number between 0 and weight-max and see "where it landed" to get the alternative passive skill that will be added to / replace the original passive skill on the tree.

Imgur
#

But I was made aware that I might not be allowed to share more information here due to reverse engineering and ToS rule

#

And the "roll a number" part uses the initialized twister state to do that.

slate rover
#

Ah oke interersting, thanks

#

this stuff seems a bit iffy with ToS for sure 😅

pseudo ocean
#

Also there is some additional parameter I don't quite understand yet.

#

Each jewel seems to have an additional hidden seed, or version or sth.

#

that gets factored in the result, and if that value is 0 my findings so far work out.

#

But that isn't always the case, so idk

#

Also I'm not quite sure if this value is really "attached" to the jewel itself or rather to that replacement stat.

#

In memory it at least is structured together with the jewel information but that doesn't have to mean anything.

worthy cape
#

When they were new there was a lot of network traffic observed when slotting in jewels and a base assumption that pretty much everything around them was communicated that way.

#

Not much to look at otherwise while still staying remotely in ToS land.

pseudo ocean
#

I also thought that this has to be set server side and the server only communicates the stat id that will add to / replace the passive skill but apparently not

#

This is for example the PRNG state for these inputs:

Input (Seed): 16563 (0x40B3) | Lethal Pride
Input (Node): 20528 (0x5030) | Instability
unknown.png
# 
0x00 unknown[0x04]
0x04 internal_prng_state[0x10]
0x14 version[0x04]
0x18 tree_node_id[0x04]
0x1C jewel_seed[0x04]
grizzled seal
hazy fiber
#

based on what he said later on, its probs not that

#

and imo version num probs hasnt been incremented based on what mark has said as well

grizzled seal
#

I mean he did describe it as a stat on the jewel that might be given a description line in the future if necessary, I'm not really sure what other stat they could possibly have hidden on timeless jewels

civic crane
civic crane
#

So you want to extract rewards, why do you aggregate all texts by mark type, instead of parsing line by line. First line type of reward, next attributes

velvet fog
#

patch 3.18.1c added 32 bytes ggdh in each directory

worthy cape
#

That sounds fun, some sort of integrity mechanism?

#

Would be unfortunate if Mr. Smoother goes out of business ^_^

worthy cape
pseudo ocean
#

I am sorry to ask a question here i am not a tool developer or in general smart in PC stuff but I'm facing a problem with the poe game itself. The problem is the latency, I have a SSD and before this problem happened the loading screen time was about 1-3 secs, I use WiFi and I thought playing on the steam might change something so I just downloaded it yesterday and it took around 30mins to download the entire 29gb game. I live in India and it doesn't have a server so I'm playing on Singapore server which is the best for me in the past atleast but now the latency on a loading screen is always 0 for 2 months now (I quit inbetween) and it stays that way until it gets up some number then the loading only takes again 1-3 secs but to get some number on the latency is taking some time now, I have tested and many time this happened the latency will stay 0 and it drops me to the login screen again. Please can anyone help me with this?

#

Sometimes 3 mins to get the latency up

#

But it always is in 0 on a loading screen

#

So it's taking a long time to load now

#

Once it's done everything is just fine

hazy fiber
pseudo ocean
#

I have exempt poe folder from my antivirus software is that what I have to do? Or something else

formal kraken
pseudo ocean
#

Ohhh

#

LoL see I'm not a tech guy at all 🤣 🤣 🤣

#

I'll reserch my antivirus software a bit more trying to figure out if it's blocking poe it a internet security software so there is a possibility it might be blocking poe

#

But this was not an issue every before

#

I'm playing this game from Jan and took 1month break only

#

Never faced this issue

long sky
#

What A/V are you using?

simple ravine
frank drift
#

guessing h is hash?

simple ravine
#

opens poesharp solution

#

iirc that's already in there for each dir

frank drift
#

and @pseudo ocean your issue may well be the integrity check GGG added yesterday

#

lots of reported issues with it

#

or at least with the patch yesterday, no confirmation the issue is actually the integrity check

simple ravine
#

wow, it's a long time since I opened this project

pseudo ocean
simple ravine
#

perhaps I should rewrite it again 😂😂

pseudo ocean
#

I have tried everything I could like installing the game again, running the hash file( the name might be wrong here but it was in my installation folder which checked and fixed poe client), turning off my internet protection (it's called firewall maybe?), Singapore VPN, installed on steam so basically I have installed the game again and again for 4 times and one entire windows installation again, Upgraded SSD, deleted caches, etc

worthy cape
#

Whole bunch of files named just ggdh in many (every?) folder in the GGPK, containing 32 bytes of textual hex digits, so 16 bytes of real data.

simple ravine
#

interesting, and not present in Steam meaning they're not actual files, hmm

#

probably some hash of some sort I presume?

pseudo ocean
rapid pagoda
frank drift
hazy wigeon
pseudo ocean
hazy wigeon
pseudo ocean
#

Ah yes, Glorious Vanity has that - this is a difficult ones, also has 2 notables that can have random stats, don't know exactly how that works yet

pure fox
#

Oh behalf of the entire POE community....

frank drift
#

bravo for breaking the TOS?

vapid pulsar
#

@pseudo ocean I was working on exactly the same thing with the jewels earlier this week. Werid coincidence.
You seem to have figured out the PRNG. Did you rewrite it by hand from the ASM? Seems like Mersenne Twister  but not quite from what I could tell

pseudo ocean
vapid pulsar
#

Yeah just as I thought. The constants where very similar.

rapid pagoda
frank drift
#

quite

rapid pagoda
#

power move: reimplement the algorithm in JS and write a patch which implements it in the online tree viewer

lunar crypt
oak estuary
#

Better make sure you pay that extra 1e-12 Scroll Fragment

rapid pagoda
#

Wisdom Fragment Fragment Fragment Fragment...

velvet fog
frank drift
#

localidentity farming karma

carmine merlin
#

Some guy made that post 10 mins after I tweeted about it. They work fast

violet path
#

too excited for ggg reply can't sleep

vapid pulsar
#

I am curious if GGG will patch it to be server side or just let us have it.

It would not be small effort and it sounds like they are very thin with developers with Poe 2 & the next league

mortal bone
#

@carmine merlin you guys might want to be careful about using decompiled code (even reimplementations of it)

carmine merlin
mortal bone
#

That is pretty heavily against tos

#

Did they allow it?

carmine merlin
#

Haven't responded yet. Only sent it about 30 mins ago

#

even if we don't use the code, someone can just create a massive table of all the notables and the effect each seed has on them

mortal bone
#

To each their own, it just feels wrong to me.

shadow mortar
#

It's getting nuked soon anyway, cool stuff but really, don't get excited lmao.

It goes against the design of the jewels, it's gonna be gone faster than wormblaster

vapid pulsar
#

When GGG released the bundle storage update I used reverse engineereing to figure out the hashing. Never had any repercussions from that.

Though this might be different.

velvet fog
hazy fiber
#

Mersenne Twister has been known for over 3+ years, wouldnt be surprised if other people had it already as well

frank drift
#

if that guy did it, other people absolutely did it

rapid pagoda
# shadow mortar It's getting nuked soon anyway, cool stuff but really, don't get excited lmao. ...

It goes against the design of the jewels
I'm not convinced it does, actually. There are still just as many unique tree permutations as there were before. Players could already see what a specific jewel does by testing it, and record the results for a bunch of seeds. (In fact, someone posted on Reddit a few weeks ago about having done this for a bunch of Elegant Hubris jewels.) If anything, this levels the playing field by making it possible to trade jewels with knowledge of what they do, instead of having to rely on streaming and/or promises

late schooner
#

#make tft far less necessary

velvet fog
#

can the above javascript code be discussed here?

mortal bone
#

Looks like the pastebin expired?

velvet fog
mortal bone
#

You are missing all the important bits from the wasm side of things

velvet fog
#

missed the most important part

worthy cape
#

Left as an exercise for the reader.

formal kraken
#

I cant believe this is actually client side

vapid pulsar
#

Started on getting it into a proper skill tree.
Got the code into WASM for Emmitt's web one, but I need some sleep.

formal kraken
#

huge

pseudo ocean
#

oh nice