#tooldev-general

1 messages · Page 4 of 1

worthy cape
#

That's XNView MP on my extracted file tree.

formal kraken
#

ah you have them extracted

civic crane
#

don't use 4k version tho, they're different

worthy cape
#

But yeah, squint at the files and see if you can backtrack them to UIImages1

velvet fog
worthy cape
#

Art/2DArt/UIImages/InGame/ItemsHeaderUnique*, I reckon.

formal kraken
#

Yeah that's the ones, trying to get them out now

#

I also figured out why I couldnt find them, because my editor properly decoded the text (utf-16) but the search didnt match because it searched with utf-8 -_-

#

thanks Kate

worthy cape
#

Yeah, just had the same with both glogg and ripgrep.

rapid schooner
#

Hey, I'm new there, is there any actual parsed ggpk data in a json format (interested in weights of gems, fossils, essences etc.)?

worthy cape
#

RePoE was the go-to place for most of that kind of data, but it's currently not maintained and stale for 3.20.

oak estuary
#

poe-dat-schema is probably your best bet now

formal kraken
#

That took way too long

image.png
rapid schooner
#

Tried to parse ggpk file using RePoe - getting this error, what am I doing wrong? Path to ggpk is correct

image.png
worthy cape
#

It needs a PyPoE with updated dat reading logic and some spec updates as well as adaptation to some of the things that changed in 3.20.

#

I hacked it into working with the wiki PyPoE fork but it isn't meant to support external consumers and I can't step up to maintain RePoE.

rapid schooner
#

Alright, thanks

velvet fog
#

@worthy cape what command you used to convert the map image to red and combine it with the base image?

worthy cape
#

Oh wait, wrong section.

#

It used to use ImageMagick to do the colorization and compositing in the past, but we got rid of that dependency as they didn't have BC7 support and it was a chore to install.

#

(magick now has BC7 since late 2021)

velvet fog
#

what command you used in imageMagick?

    convert Abyss.png -fill red -fuzz 50% -opaque white AbyssRed.png
    convert Base16.png AbyssRed.png -gravity center -composite AbyssBase16.png
worthy cape
#

Here I used colorize rather than a straight-up multiply to make the glyph stand out better like it does in the game.

#

Git history suggests this:

f'magick convert "{ico}" -fill rgb({color}) -colorize 100 "{ico}"'`
f'magick composite -gravity center "{ico}" "{base_ico}" "{ico}"'
rapid schooner
#

What's the best way to get offers from trade site? Let's say, I want to get all fertile catalysts offers with stack size 10+ and their prices. The most stupid and straightforward way is to hardcode query and then scrap data from page but I need something more optimal.

civic crane
#

@rapid schooner how you will use this data, have you looked into poe.ninja api

frank drift
rapid schooner
rapid schooner
frank drift
civic crane
rapid schooner
rapid schooner
frank drift
rapid schooner
oak salmon
#

Couple questions about PublicStashChange API:

  • I can't seem to find any indicator of when the change was made. can I only know the age of listed itens if I'm processing in realtime? can I tell if I'm "catching up" or falling behind the current changes (particularly given rate limiting)?
  • Is the PublicStashChange ID supposed to repeat if data from the same stash tab reappear? If an item is added to a tab for example, or a note changes on an item. I understood from the docs that if a tab was unlisted it would would only include the ID (so I can unlist the itens that were on it), but I've been running the stream for a bit and haven't ran into any repeated stash ids.
civic crane
#
  • theoretically the delta change between your current and next id (more specifically some average over N requests) should tell catching/falling. alternative is to query https://www.pathofexile.com/api/trade/data/change-ids and see how far you are (this id is realtime, not 5-min delayed)
  • yes, they repeat
oak salmon
civic crane
#

@oak salmon 1) their backend has 5 shards, you can compare pairwise 2) it will take 5min for test but you will be sure at least, add condition in code to find stash by your accountname, then change stash in the game, wait, compare ids

#

If I see the latest how can I tell how far am i?
you can't. If you have data on the rate of change, you can approximate a bit, say normally it's N changes per 24h, and you are at (REALTIME_ID - YOUR_ID) === N/2, then currently you're reading stashes that people last touched 12h ago

oak salmon
rigid oxide
#

hmm, does anyone know if new api tool applications are active atm? sent a mail last monday, didn't get a response yet

oak salmon
#

mine took about a week, got it 5 days ago

rigid oxide
#

ah that gives me hope, ty

pliant cipher
#

Why are projects never easy, I was looking into extracting ggpk stuff for a .Net project, but LibGGPK2 doesn’t support steam files and LibGGPK3 doesn’t support bin64 files as well, so I figured I’d look into just calling PoB’s static data lua and converting it to json… it works but the format is totally wonk and I’m having to re-parse the key value pairs that lua gives back from its LuaTable format

hazy fiber
#

lua index 1 is strange, but other than that it shouldnt be too bad?

#

also I think you can just get PoB to output it as a converted json

worthy cape
#

oO(make C# interop bindings to Bun)

#

What amount of data access are you looking for? Just bundled file access, DAT64 scaffolding, specs for those?

vestal pond
rapid schooner
#

Getting response 403, what am I doing wrong?

image.png
formal kraken
#

There is a user agent format described here: https://www.pathofexile.com/developer/docs which you might be able to match. Not sure about the details, havent messed with the trade api myself and for everything else I do have an oauth token

rapid schooner
pliant cipher
rapid schooner
#

What's the limit of trade-site api calls per minute (or hour)?

worthy cape
#

Have you looked at the x-rate-limit-* headers of the responses?

rapid schooner
#

Thank you, now i see

cursive dome
#

Hi, could anyone tell me where I can find information about hashes_ex? I'm trying to understand the relation between hashes_ex and regular node skill hash so I can get cluster jewel passives allocated when I import my char to pob on my personal side project. thx

worthy cape
#

Not sure if they've been documented on the forum somewhere but you might be able to squint at the PoB source code to infer their meaning.

desert basalt
#

!tabs

kindred pastureBOT
#

Stash tab sale when? Generally, stash tabs are on sale every three weeks starting the second weekend of the league - the subreddit has an estimated stash tab sale day in the sidebar

How do I get points? Purchasing the First Blood starter pack is recommended, as it has 200 points, a stash tab, and a weapon effect ||https://www.pathofexile.com/purchase#FirstBloodPack|| Alternatively, supporter packs are more cost effective than individual points, and give you extra stuff too.

Which ones do I get? Recommended first tabs are:

  • Currency Tab
  • Upgrade to Premium (easier to list your items for trade)
  • Map tab (after finishing the campaign)

_ And then what?_ If you're focussing on something specific eg. essence farming, picking up the essence tab might be a good investment. The same goes for the divination, metamorph, and other specialised tabs. However, you may find you can handle things with a combination of regular tabs or a quad tab (and the extra storage space doesn't hurt either)

worthy cape
# desert basalt !tabs

You may want to run commands in a channel more appropriate for it, this seems off topic for tool development.

desert basalt
oak estuary
#

Fwiw, PoB actually does cluster jewel allocation wrong when importing characters. There are edge cases where allocated nodes in-game don't get allocated in PoB

cursive dome
rapid schooner
#

Did anyone try to make requests to tradesite api using proxy?

frank drift
#

why

velvet fog
#

how is extended hashes being used? I got this

    [hashes_ex] => Array
        (
            [0] => 0
            [1] => 1
            [2] => 5
            [3] => 7
            [4] => 9
            [5] => 10
            [6] => 11
            [7] => 12
            [8] => 13
            [9] => 15
            [10] => 16
            [11] => 30
            [12] => 32
            [13] => 33
            [14] => 35
        )

from https://www.pathofexile.com/character-window/get-passive-skills?accountName=Mathil&realm=pc&character=PrettyLimitedPowerTbh
but had no idea and the website https://www.pathofexile.com/developer/docs/reference#extra don't have any explain
also, they have empty extended hashes from url if you click passive from character https://www.pathofexile.com/fullscreen-passive-skill-tree/3.20.2/AAAABgMCbQECBLMFQgj0EFEQWBEtE3EVVhZvFy8YXRo4HMQc3B0UHxgfxyBuJKonLyt4LJwtfTD4Nuk6WDrYOz88BT77QXJBlkWdSi5Kn0yLTLNQMFH7U1JVxlXWWK5Z816lZOdmVGaeZ3FqG2pDa9t2EXyDgKSCm4PbhXuGyom8jmSOvo8aj0aQVZD6ksGa4JtRm6Gg5qHHogCnCKelqyqt8a6zrv-1BLb6uF27AbyavOrAZs0W0B_Q9dLF1AfVi9Wm217cjd2M323fsONq6-7sOO9L73zv6_Af96b60v75AAQnAj77uhqhx5Y2qyruqbhd?accountName=Mathil&realm=pc&characterName=PrettyLimitedPowerTbh

velvet fog
#

does anyone know the boundaries for the monster resistance icons shown under the health bar?

pallid wraith
#

Literally whether it's greater or lesser than 0

#

At 0 there's no icon

#

Same as the old text based display

#

You can confirm this in pvp if you're uncertain

hazy fiber
pallid wraith
#

Right, yeah

civic crane
#

umm, all enchants from heist are removed, what does Tempering Orb do now? any patchnotes

hazy fiber
#

are they? I havnt seen anything around that

civic crane
#

first thought smth wrong w my code, then checked and no such stats on trade too

hazy fiber
#

I know they changed the descriptions in 3.20.2, would be odd to remove them so soon after

#

and heist enchants definitely existed earlier this week

#

so either unannounced change, or the wording change messed you up

hazy fiber
civic crane
#

right, ty

hazy fiber
#

np

civic crane
formal kraken
#

Anybody know how foiled items in ItemVisualIdentity are handled?

civic crane
#

isnt it just a shader

formal kraken
#

oh that would make sense

#

I was assuming alternate art

worthy cape
#

@formal kraken The new foils are gradient textures with some shaders in Draw2D.hlsl.

#

(there's one or two more that are unused or for other content)

formal kraken
#

How does the wiki generate the foil image version? Or is it grabbed from the CDN?

worthy cape
#

I don't think we do much in the way of visuals. No idea about old foils and the new ones are just tabulated here with the gradient images as a hint.

formal kraken
#

oh mh that's an old foil

worthy cape
#

Art/2DArt/Lookup/Gradients/*.dds for the gradients

#

I suspect that the current effects are probably the same old Draw2D logic.

#

Using the same thing for both the tooltips and item images.

#

Didn't get around looking into any captures as I haven't blown my key yet.

formal kraken
#

kk thanks, I am beginning to think this doesn't matter anyways for my usecase

median marsh
#

just wrote a rushed hacky CLI tool to find "usable" timeless jewels in your stash tabs in a fit of madness from running a bunch of 5-ways and having no idea how to sell them

#

seems like I can't share images yet sadly

craggy beacon
#

that sounds cool

median marsh
#

still need to figure out how to handle glorious vanity small passives

#

not sure if there's a better source out there or whatever PoB Community uses

grave wren
#

hm is there any tool still online that tracks unique stash tab contents? Cant seem to find any

worthy cape
#

Vague memories of that it's one of the things that is annoying or impossible to get at through the API.

grave wren
#

yeah i think the one i remember used the trade api to track it

craggy beacon
#

They treat each unique category as its own stash tab, so if you include it in a snapshot it eats up a lot of your ratelimit to request each sub tab

grave wren
#

😦

craggy beacon
#

Same issue with map tabs, you can track them but only if you are doing the tracking in way that it's okay if a update takes like 30 mins

velvet fog
hazy fiber
median marsh
#

current scope is however many can fit into N account stash tabs before GGG pulls out the ol' rate limiter on me, so this looks promising

#

Readme does perplex me a bit, referencing some .json files that seem to not exist (Jewel_Node_Link.json) or assumed I somehow got a copy of (alternate_passive_additions.json)

hazy fiber
#

yeah the LuT only stores the actual numerical outputs of the prng, I think we got rid of Jewel_Node_Link.json at some point because it went out of date when ggg shuffled some nodes around, you can use whatever method you want but I should probs include an updated one in that repository when I update them for next league,

PoB has its own internal one, its basically just what nodes are in range of what jewel

and alternate_passive_additions.json and alternate_passive_skills.json both come from the ggpk (theres one included with the generator but its out of date for curse nodes from GV)

#

keshes tool is popular I would be surprised if it didnt have GV small nodes in its csvs, and it might be a bit easier to work with, the LuTs I linked are only needed if speed is the biggest concern

#

ahh the "all csvs" are only 21mb total, where GV compressed byitself is like 19.2 with the minimal data so maybe his doesnt include those

median marsh
#

gotcha, thanks for pointing out where those files went! I'll probably switch over to that dataset once start implementing Glorious Vanity value stuff (aka once I figure out why people actually use them).

#

and it'll also force me to refactor my current memory hog situation (I just glug all the seed data in-memory and balloon RAM usage to 1.5GB lol)

grave wren
velvet fog
#

PoEDB (https://poedb.tw/us/)

A number of improvements have been made to the PoEDB website recently, including

  • The content width has been increased from 1140px to 1320px, and you can see more content on a wide screen
  • All fonts increased by one level
  • Added level chart for Gem skills
  • When the unique item has damage/defence affixes, the above attributes will also be modified
  • Added the display of the Ascendancy tree
  • Added the Passive title background image
  • Added the Passive Mastery title background image, and the display of affixes is beautified
  • Added the Monster title background image, the resistance icon, and the area information
  • Added the Quest title background image
  • Added the Atlas location display
  • Added the Chapter location display
  • Added the The Pantheon location display
  • Separation of general Bestiary Recipes and HardMode Recipes, rewrite the Recipes
  • Added the Bestiary image
  • Added the Map Items background
  • Modified the list layout of many pages to two columns to make mobile browsing smoother
poedb
PoEDB, Path of Exile Wiki

PoEDB provides things come out each league, as well as items, uniques, skills and passives. Path of Exile Wiki editing functions.

stoic locust
craggy beacon
#

It might have to do with the public stash stream, if the whole thing was one tab any change would mean the whole tab would be written to the stream which could be a lot of data for some players (this would be even more impactful on the map tab which is the first tab they created that functions this way). Probably just more efficient on their end to work with smaller stash tabs but it does have some downsides on the API side

stoic locust
#

The live search for trade?

craggy beacon
#

I'm not sure how trade works internally, but the way it works for third parties is they have a stream of public stash tab changes. When anyone in game makes a change to a stash tab that is public the entire tab not just the changes is written to the stream so third parties can update their information on listings/whatever they are doing with the data. The more items in a single "tab" (or sub tab) the bigger each of these updates will be. So splitting things into smaller sub tabs will cause smaller updates.

stoic locust
#

Yikes

worthy cape
craggy beacon
#

I don't have any information on how they do internal stuff but it would make more sense that they didn't come up with this concept just for external use and the stream is used internally as well but who knows

worthy cape
#

I believe it's mostly tied to the UI with the buttons to change tabs and the lazy loading of each tab inventory.

stoic locust
#

I mean ... item enters stash tab, normally id expect there'd be something that gives just that specific item and what tab it entered etc

worthy cape
#

For both map and unique tabs, filter highlighting is mostly smoke and mirrors based on static or cached data.

stoic locust
#

So theoretically it should be possible to just push the new item or changes in the stash to the stream

worthy cape
#

Even for quad tabs vs. regular tabs, there's a noticeable difference in load time.

stoic locust
#

Without needing to push the whole stash

worthy cape
#

The trade river is very fun indeed. One good reason for pushing atomic tabs is that you always have a full view on a tab. There's no good point to start if all you have is deltas from the dawn of time.

#

You could envisionably have some mix of keyframes and deltas, but that makes things more complicated and might be more state to track both on sender and receiver.

#

"something changes in a tab? pull whole state and schlep it out on the wire"

craggy beacon
#

There is a way to do it for sure but this way does have some advantages. It seems more fault tolerant if you miss some updates or want to skip ahead to catch up you might have a few duplicate items/unlisted items hanging around for a bit but if they update any item in that tab everything will be fixed

stoic locust
#

Well it wouldn't just be deltas.

There would still be periodic updates for the full tab

craggy beacon
#

Plus I don't think GGG themselves need the deltas for anything so it would be extra work on their end

hazy fiber
#

my guess is internally tabs were saved to the database as a full tab at a time, then when they made public tabs they just mirrored that stream over to the
"river" filtering out any non-public tabs

#

this also makes sense from the public stash perspective, rather than having to keep track of potentially hundreds or thousands of changes since the tab was last "saved" you just push the current state of said tab, dont need to actualy track what happened in the tab just how it ended up

craggy beacon
#

Yeah and while this way isn't as intuitive right off the bat after working with it for a bit I think I prefer this approach

frank drift
#

Anyone able to log on to poeplanner successfully? Doesn't work for me.

civic crane
#

less AS but higher single hit, what if they're slowing down the game, so servers can be run at 20 tick rate in poe2 marauderthinking

median marsh
#

trying out VSStudio's JSON -> C# feature on the passive skill tree, maybe not the best idea:

image.png
graceful barn
#

tooldev adjacent question, I have a PR that I manage for aurabot export functionality in PoB. I have another dev who wants to add a commit to the PR, but I can't seem to find the button to add him as a contributor

#

heres the PR in question

#

just not sure what hoops I have to jump through to allow him to push a commit

worthy cape
#

IIRC, it's on your own repository.

#

Normally there's a bit of inheritance from the parent repo somehow, but I'm not sure about the particulars there. It's always been a bit confusing.

graceful barn
#

alright, I think I did it right if thats the case then

worthy cape
graceful barn
#

does seem a bit more convoluted than I expected

worthy cape
#

There's the option to allow maintainers access, which you set on the PR itself.
Then there's explicitly adding collaborators to your whole fork.

graceful barn
#

Thanks!

untold trout
clever thistle
worthy cape
#

Here's hoping that I didn't break anything in ooz 😉

hazy fiber
formal kraken
worthy cape
#

Nothing since I added SIMDe for cross-platform SIMD. Probably.

#

Heh, I was going to test the node tool linked above in a container VM but got royally sidetracked by that my IPv6 suddenly not routing.

formal kraken
#

I did not expect you to reply so fast templarLul

worthy cape
#

Suitably deep in the rabbit hole, I finally got my network fixed so I could actually install the software:

RuntimeError: abort(RuntimeError: abort(CompileError: WebAssembly.instantiate(): Compiling function #3 failed: Wasm SIMD unsupported @+582). Build with -s ASSERTIONS=1 for more info.). Build with -s ASSERTIONS=1 for more info.
    at process.abort (/usr/local/lib/node_modules/pathofexile-dat/node_modules/ooz-wasm/build/ooz.js:9:6498)
    at process.emit (node:events:512:28)
    at emit (node:internal/process/promises:150:20)
    at processPromiseRejections (node:internal/process/promises:284:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:32)

Node.js v19.6.1
#

Guessing that whatever the Docker container node:latest is doesn't have the cool extensions enabled or something about CPU capabilities that doesn't carry into the container.

formal kraken
#

sounds like it compiles to wasm with simd but runs on a platform without simd

worthy cape
# 
bool CpuFeatures::SupportsWasmSimd128() {
#if V8_ENABLE_WEBASSEMBLY
  if (IsSupported(SSE4_1)) return true;
  if (v8_flags.wasm_simd_ssse3_codegen && IsSupported(SSSE3)) return true;
#endif  // V8_ENABLE_WEBASSEMBLY
  return false;
}
#

Oh for crying out loud.

formal kraken
#

but that's runtime?

#

ah nvm the #if

worthy cape
#

The outer VM was configured with the default qemu CPU type of kvm64 which is a very conservative setting to make it possible to migrate VMs between cluster hosts.

#

Something more akin to host would be preferred here.

formal kraken
#

why is this relevant for node/wasm?

worthy cape
#

Based on the error message, ooz-wasm uses SIMD, which doesn't seem to have any form of scalar fallback path.

formal kraken
#

but it's compiled to WASM, if V8 has WASM SIMD enabled, it should run no matter if the CPU supports it nor not?

worthy cape
#

Deep in the guts of function-body-decoder-impl.h this seems to be run in read_value_type, I assume that the code is using those wide registers and it's a hard requirement that the platform has SSE4.1 or possibly SSSE3:

    case kS128Code: {
      if (!VALIDATE(CheckHardwareSupportsSimd())) {
        DecodeError<ValidationTag>(decoder, pc, "Wasm SIMD unsupported");
        return kWasmBottom;
      }
      return kWasmS128;
    }
formal kraken
#

Ah I see, I thought the error was coming from V8's instantiate not from running any code

#

aka it read simd instructions without simd support

worthy cape
#

I guess it's raised from when it's parsing the WASM bytecode itself.

formal kraken
#

oh function-body-decoder-impl.h is V8/node code not ooz ..

worthy cape
#

Yeah, that was it. Enabled a cooler machine type on the VM and now I'm like OP:

RuntimeError: abort(TypeError: Cannot read properties of undefined (reading 'filter')). Build with -s ASSERTIONS=1 for more info.
formal kraken
#

should go back to bed

worthy cape
#

The library has a whole bunch of unguarded statements like const files = config.files.filter(path => !isInsideSprite(path)) which will blow up unceremoniously if you're missing that key.

#

(I have no idea what I'm doing in this ecosystem, it's very foreign to me)

civic crane
#

Just woke up and there was a huge investigation templarLul updated, run npm update --global pathofexile-dat

worthy cape
#

Nobody expects the Templar Inquisition.

#

(seems to work)

fathom tree
#

Hey there, quick question since I cannot find the info myself. Is there a way to get quest progression data for a character? Just general info about what quests are active for the char and stuff like that.

civic crane
fathom tree
naive haven
#

Hey, this is kind of niche, but is it known that summon phantasm gives attacks the spell tag in the context of certain supports?

#

this setup isnt working in pob, even tho it works ingame for whatever reason

image.png
#

atm i've only found faster casting to work, but haven't done much testing

#

Also, the cast speed > attack speed mods don't stack in PoB, even though they do ingame. I think this problem also exists for the spell damage > attack damage mod

carmine merlin
naive haven
#

ye

#

im getting 30% attack speed from the lvl 1 faster casting on the stat sheet ingame

carmine merlin
#

Interesting, I'll look into fixing it

#

Interesting that Summon Phantasm would grant it the spell tag cause it doesn't add that tag from the gem

requireSkillTypes = { SkillType.Attack, SkillType.Damage, SkillType.CreatesMinion, },
addSkillTypes = { SkillType.CreatesMinion, SkillType.MinionsCanExplode, },
excludeSkillTypes = { SkillType.MinionsPersistWhenSkillRemoved, },```
naive haven
#

maybe the faster casting is being applied by .createsminion?

#

other support gems w/ the "cannot support skills that creates minions" don't work with it, like pinpoint

carmine merlin
#

Faster casting has these tags

addSkillTypes = { },
excludeSkillTypes = { SkillType.Instant, SkillType.FixedCastTime, },```
naive haven
#

hmm

#

need mark on this lol

carmine merlin
#

I'll ask someone at GGG

carmine merlin
#

@naive haven confirmed Wilma's shouldn't stack with Anom Kinetic Bolt and it being linked to Summon Phantasm and Faster casting is also a bug

naive haven
#

first part is lame

#

second part makes sense

hazy fiber
#

or as local said, is bugged

naive haven
#

yeah

#

another oddity, when using the same setup with minion speed support instead of faster casting, spiritual command allows the attack skill to benefit from minion speed

#

which makes sense i suppose but is a little strange

#

looks like pob recognizes this so ig its intentional

trail depot
#

Is there a way to speed up PoB's "Show Node Power"?

hazy fiber
#

do you have full dps turned on?

trail depot
#

No. Eff Hit Pool

#

All of my skills are disabled as well.

#

Something else that might help would be being able to run in the background. I've been tempted to look into adding it myself but I'm pretty dang unfamiliar with lua.

trail depot
hazy fiber
#

having full dps turned on requires it to run a full pass for each skill in full dps AND one for main skill, so thats one of the main things that slows it down

#

it also seems to slow down if you leave it open but have been unable to determine why, as the memory leaks have been fixed

trail depot
#

Full DPS is worse for active-skill heavy builds then I assume?

hazy fiber
#

as for run in background, it use to do that, but the main thread is paused now, its possible to run it on another thread that wouldnt get paused, but think that might cause issues, the pasue in background probs needs to be changed at some point anyway

hazy fiber
#

not sorting by full dps, just having it turned on

trail depot
#

OH. Interesting.

#

I did have Full DPS on, but wasn't sorting by it.

hazy fiber
trail depot
#

(Though I later disabled all of my skills)

hazy fiber
#

I think the fps limiter is on th c++ simple graphics side, not PoB itself

#

so unsure if I can actualy do much about it

trail depot
#

Honestly, even a point in the right direction might be helpful. I'm a programmer, but, I code in C# and Typescript for work; C and C++ on the side.

#

Is the fps limiter what determines whether PoB is running in the background?

hazy fiber
oak estuary
#

@worthy cape has been working on somewhat of a rewrite of SimpleGraphic, but afaik it doesn't do anything regarding background tasks, just OpenGL issues

hazy fiber
#

only background task I know of that runs while alt tabed is downloads, but some of the other things on thier own thread probs runs as well

#

not that many of those exist

trail depot
#

Is line 360 of ui_main.cpp what's doing it?

hazy fiber
#

as far as I know yes

trail depot
#

Seems simple enough to change then. Would just need to figure out how to add an option for it in the frontend. 😛

hazy fiber
#

but simple graphics changes are super uncommon for us to push through

trail depot
#

Understandable. I noticed there were like 80 PRs so I figured the team is pretty careful.

hazy fiber
#

I think the better solution is fixing that PR I linked, and just limiting the fps to like 4 while in background, that way it can still run background tasks like node power

#

the forground limit was the main issue iirc

hazy fiber
oak estuary
trail depot
#

Even a feature like, "Allow Node Power Calculation in Background" would be useful to me. Because then it would begin limiting CPU again once node power has finished calculating.

worthy cape
#

WTB working brain, then you all can have all the features.

#

On background tasks, as some of the frame pacing inhibits the IPC to/from subscripts it could be an improvement to have a healthy little wait and process subscripts periodically as their callbacks to things like ConPrintf would still tick.

outer jay
#

Is there some way to get a testing oauth2 setup without going through the email process? Trying to mess around locally to see if my setup works before going through the process

velvet fog
grave wren
compact isle
lone palm
#

Hi! I just downloaded awakened poe trade, and i can't edit the map mods check. I saw there was an issue that has been fixed in a recent comit on github, but it has not reached a released version yet. How can i quick fix it ? (It seems to be linked to profile file generation)

frank drift
lone palm
#

Ok i managed to fix it by uninstalling it, deleting manually all config files in %appdata%, reinstalling an old release, and let the auto update function update to last version (manual update bricked the config file too)

naive haven
#

@carmine merlin is blade trap meant to work with the attribute stacking mods? tried in pob and it's not adding them

#

oh ye it does work, just doesn't show up in the added min/max section of the calcs tab

carmine merlin
naive haven
#

doesnt show up in here

image.png
brave venture
#

monkas dont judge the quality of that pob

carmine merlin
#

It's cause the mod has with this weapon in it, that it isn't showing up in the breakdown

naive haven
#

wack

clever thistle
#

autohotkey can't tell which monitor the current window is on? (or do i just reeeeealy suck at google?)

#

oh now i find it lol, thanks everyone! 😁

modern snow
#

For the trade-exchange-request-limit rate limit policy, the Ip limits are 7:15:60, 15:90:120, and 45:300:1800. However, exceeding the 15:90:120 limit results in being timed out for 300 seconds instead of 120 seconds.

Is it worth sending in a bug report or someting? 🙂

formal kraken
#

Does every base consistent of exactly or at least 2 words?

#

Some shields have 3 words, but everything else seems to be 2 words

hazy fiber
#

"ring"?

formal kraken
hazy fiber
#

"Chestplate" (body armour)

hazy fiber
formal kraken
# hazy fiber kalandra ring

okay, I need this 'heuristic' to only parse magic items properly. I assume the only way to get a 'Ring' would be by bricking it and vendor recipe?

hazy fiber
#

also claws (Awl/Blinder/Gouger), daggers (Sai/Trisula/Poignard/Stiletto), rune dagger (Skean), scepters, staves and probs more, cbf looking

hazy fiber
#

so you wont need to come across it very often, imo just check if it has ring and not one of the other base type names before it

formal kraken
#

My Problem is parsing a pob magic item like (screenshot). It has 2 mods a suffix (of ...) but no prefix. So I need to know the 2 mods are actually one ... I was hoping I could have an heuristic that the amount of words on the base never goes under 2 (or in some cases 3)

image.png
#

but I guess this is also not 100%

#

and I dont really wanna pull in an entire mod db

grave wren
#

doesnt repoe offer a list of all base items or do i misremember

formal kraken
grave wren
#

would be the most accurate way, unsure if any heuristic would work well enough

formal kraken
#

this sucks

#

thanks

serene shadow
#

Hi, anyone know how i can get the public stash data? in a way that poe ninja or excilence/poe stack get that data to generate economy charts?

frank drift
serene shadow
#

thanks!

serene shadow
#

do i need to register to fetch the economy data?

craggy beacon
#

You need to send an email requesting access to the API to get approved to access the public stash stream

worthy cape
#

The river used to be accessible in the past, indeed.

serene shadow
#

I see! thanks again haha

serene shadow
#

how long do they usualy take to grant access?

worthy cape
#

It has to reach a human but I don't think it's super slow, at least not during weekdays.

frank drift
#

generally hours but as noted it's Sunday

velvet fog
#

@compact isle News posts are missing breadcrumbs. can this information be added to know the language of the article?

image.png image.png
serene shadow
#

goddamn ive got so many ideas and they dont answer. Does anyone of you have access and could supply me with an old stashtab file so i can start implementing some evaluations until i get access?

oak estuary
#

It's been less than a day, lol. Currently 2:30 in the morning for them, also

formal kraken
#

Documentation not enough? https://www.poewiki.net/wiki/Public_stash_tab_API

Public stash tab API

Public stash tab API is a basic JSON interface to obtain information relevant for general trade using Premium Stash Tabs for PC clients.
Basically the interface serves global update information (considerable as an update-"stream") on any public stash tabs recognized by the game servers.
API doesn't contain any historical data and always returns ...

serene shadow
#

Yeah i totaly get it. Absolutely no blame to them. Even if it takes days it would be fine. Its just that i have time today and would like to get startet 🙂

#

Some old example data would be nice to try out ways of how to implement things

#

as far as I have seen there is none available online ?!

formal kraken
#

might also get lucky with google cache

serene shadow
#

might be I am just too dumb, but i can't see a way to get the json data from this

#

Sorry I am quite new to this..

grave wren
#

click on one of the longer links with a uuid, select the timeframe it captured

image.png
#

then click on the blue dots to get it

serene shadow
#

Thank you very much!

#

All of you 🙂

formal kraken
#

good luck with your project

serene shadow
#

Thanks ill let you know if I manage to do anything interesting

pliant cipher
#

Hey yall, for those who might know, is the stat_translations file format in the ggpk/steam bundles file for translating stats in a standard file format or is the format something GGG made? I'm trying to wrap my head around the standard, especially when looking at stats with different options

#

it seems... kind of straight forward, but then I took a look at the timeless jewel translation stuff and it's... interesting

worthy cape
#

Very custom as far as I know.

pliant cipher
#

Like this is pretty straightforward to parse:

description
    1 local_maximum_prefixes_allowed_+
    2
        -1|1 "{0:+d} Prefix Modifier allowed"
        # "{0:+d} Prefix Modifiers allowed"

"Two possible values for local_maximum_prefixes_allowed_+, if -1 or 1, it's the numver formated with string, else use the other"

#

Then you look at the timeless jewels...

description
    4 local_unique_jewel_alternate_tree_version local_unique_jewel_alternate_tree_seed local_unique_jewel_alternate_tree_keystone local_unique_jewel_alternate_tree_internal_revision
    20
        1 # 1 # "Bathed in the blood of {1} sacrificed in the name of Xibaqua\nPassives in radius are Conquered by the Vaal" reminderstring ReminderTextConqueredPassives
        1 # 2 0 "Bathed in the blood of {1} sacrificed in the name of Zerphi\nPassives in radius are Conquered by the Vaal" reminderstring ReminderTextConqueredPassives
        1 # 2 1|# "Bathed in the blood of {1} sacrificed in the name of Ahuana\nPassives in radius are Conquered by the Vaal" reminderstring ReminderTextConqueredPassives
        1 # 3 # "Bathed in the blood of {1} sacrificed in the name of Doryani\nPassives in radius are Conquered by the Vaal" reminderstring ReminderTextConqueredPassives
...
#

ouch

#

1 at the start is the index of the mod at the start, but... the rest doesnt make sense since the names dont change... shrugR

worthy cape
#

Some of those stats seem vestigial.

pliant cipher
#

probably.

worthy cape
#

I assume you match up the four values before the string with the four stats listed up-top.

#

There's three variations, of which the second went through two revisions, obsoleting Zerphi.

pliant cipher
#

That's what I'd assume. I forgot I can look at how RePoE's output for stats and see how they get matched up.

#

interestingly there's 16 more options after, so the four are just associated with these translations

worthy cape
#

A legacy keystone 2 is Zerphi, while a current+ keystone 2 is Ahuana.

pliant cipher
#

But there's only 4 in a group so I guess you match them up by index

worthy cape
#

1 # 2 0 "Bathed in the blood of {1} sacrificed in the name of Zerphi\nPassives in radius are Conquered by the Vaal" reminderstring ReminderTextConqueredPassives
would match something like

local_unique_jewel_alternate_tree_version: 1
local_unique_jewel_alternate_tree_seed: #
local_unique_jewel_alternate_tree_keystone: 2
local_unique_jewel_alternate_tree_internal_revision: 0
pliant cipher
#

ah

#

oh, yeah okay that makes sense

#

Welp, time to write a parser

#

thanks zao!

worthy cape
serene shadow
#

Sorry to bother again, but I cant for the life of me get the authorization to work

#

I try to use this post to generate an access token in Python. So far i have tried numerous tutorials and combinations (writing everything into the body etc) but I always get 403'd

#

Probably this is a dumb question but its the first time I am using API's and i was hoping someone might be able to help me maybe

worthy cape
#

Did you remember your user agent?

serene shadow
#

nope. Thanks, trying that!!

compact isle
serene shadow
#

I finaly got it to work! Thank you guys!

velvet fog
exotic egret
velvet fog
#

I want the article language, but I can't find any useful information in the post.

#

breadcrumbs have forumID, which is language information.

exotic egret
#

Announcement posts haven't had breadcrumbs for a long time. I don't think they'll be added any time soon.

velvet fog
exotic egret
#

Is there a reason you're avoiding the RSS feed for news posts?

velvet fog
#

RSS are announcements only, I want to know other official messages, it seems that it is better to get from recent forum posts

compact isle
serene shadow
#

Hello fellow exiles, I am back with another, lets face it: probably stupid question regarding the Public stash tab api.

For my purposes I would like to have a snapshot of the current state of the economy. Meaning, I need to get the contents of the public stash tabs once or maybe twice a day.

My current understanding is the following: the API streams only the updates, paginated by the nex_id. So if i want a "complete" set of listed Items i will need to store the listed stashes localy and update my database with each update of the stream.

formal kraken
serene shadow
#

Am I correct in how this works?

And my question would be: is there a way to just have my code running once or twice a day and acquire a snapshot, without constant updating?

worthy cape
#

The river (psapi) sends the full contents of any modified tabs.

#

You need to consume the river in full to not miss updated tabs.

formal kraken
worthy cape
#

It's all an incremental stream of full tabs.

#

There's no way (in the river) of getting the state of a particular tab without sweeping the whole stream for the last instance it was updated.

craggy beacon
#

Technically you could store the last next_id you processed and then step through them until you reach the current id when you start the process. You could run this whenever you want but it probably isn't practical because of the rate limit

serene shadow
#

CONSUME ALL OF THE STREAM

#

Thanks a lot you guys!

worthy cape
#

Catching up like Zach describes is viable, but you'll miss out on interim updates for any particular tab if it has had an update closer in time.

serene shadow
worthy cape
#

Like if a tab is updated at id=9001 and id=9005, the one at 9001 will not have any contents.

#

You should with some pipelining be able to at least keep pace and probably catch up slowly.

#

Typically you'd start at a recent point in time and never really pause.

#

There's some tricks you can do to pipeline better, like reading out the ID from the raw bytes of the response before you parse the actual response so that you can kick off the next request.

#

It's a bit of a struggle still as time-to-first-byte is often high.

craggy beacon
#

One other thing to consider is if you are working on aggregate data you really don't need need all stash updates. If you expire listings after a short amount of time, missing/skipping some updates is not the end of the world

#

For the type of items poe ninja tracks there are more than enough listings in a 2 hour window. If every 12 hours you captured 2-3 hours that would probably be fine

serene shadow
#

Thats very true, this might be the way i will approach this, as I dont really have the possibility to have my pc running and saving data all the time

#

As far as I have seen, it is not possible to get the up-to-date prices from poe.ninja right?

worthy cape
#

What kind of traffic volume do people see these days? When it was a bit backed up I got around 5 MiB/s.

#

Ah, running on a non-server device. That's fun.

craggy beacon
#

Running on a non server will make it harder to catch up as well unless you have really good internet

formal kraken
#

could just start with a current id, iirc poe.ninja has an api for the current id?

serene shadow
#

Yes they do.

craggy beacon
#

since the stream is delayed 5 mins you won't be able to pull an ID you see there until 5 mins after you've seen it

serene shadow
#

Well it seems like the little idea of evaluating the current state of the market is getting out of hand even faster than i wouldve expected. Might use some necromancy to revive my old laptop and set it up to scrape the stream for me now.
Thanks again for the help all of you!

worthy cape
#

Repurposed laptops? Highly available compute element with integrated UPS.

craggy beacon
#

PoeStack uses variable lookback windows for calculating prices based on how much data is available in each window selecting the smallest window that has enough data. These are the counts of windows used for currency pricings right now in Sanctum. So most of the time 2-4 hours was enough, but there are a few cases where it needed to look at bigger windows

image.png
serene shadow
worthy cape
#

Top machine there powers my PoE game data archival and serving project hackerman

IMG_20230306_172703.jpg
formal kraken
grave wren
#

got the bottom one i think 😄

worthy cape
#

Define R6, big storage machine and toy VM host, has ten of my disks.

grave wren
#

bottom full visible one is a meshify?

worthy cape
#

Middle one I think is a Meshify C Mini.

#

Bottom half-obscured machine is a HP ML10 v2.

grave wren
#

til they made a mini meshify c

formal kraken
grave wren
#

time to rob zao

#

dibs on the 3d printer

worthy cape
#

Noo, I just upgraded it.

formal kraken
#

I'll take the switch

grave wren
formal kraken
#

someone get the blueprint

worthy cape
#

I can't wait until I manage to get the data thing moved over to the Hetzner machine.

#

Probably going to take a while as @formal kraken baited me into thinking over if I could ingest releases faster.

formal kraken
#

oh no

serene shadow
#

I ll take the candles from the top shelve

#

love meself some ambience

modern snow
#

I was about to start a simple, for-fun project to record prices of bulk exchange items frequently enough to see differences between regions. Are there others already doing this? From what I could see, poe.ninja and poestack both don't seem to expose pricing data more granularly than daily.

craggy beacon
#

You can change poestack's graphs to hourly for the last 48 hours

image.png
modern snow
#

Neat! I might still go ahead and have some fun making something, but that's great 😄

#

Does poestack just hit exchange and trade for that data or does it also look at psapi at all? I can't imagine that looking at psapi would be worth it for this sort of thing

craggy beacon
#

it only uses psapi I make no requests to the trade api

modern snow
#

😮 I see, why did you choose to do that? Are you also making use of maintaining stash state for other things?

craggy beacon
#

I am now using the data for other things, but it was mainly about scale I need to execute around 80 thousand trade searches per hour to maintain the hourly data for everything tracked right now across all the leagues

#

it's really easy to execute those 80k searches vs my own DB since I don't have any rate limits and don't need to leave the datacenter

modern snow
#

I see. Do you worry at all about missing long-lived listings that are in stash tabs that have had no changes in a while? Or are you just always tracking since league start anyway?

craggy beacon
#

for pricing I only use listings that are less than 48 hours old and for most pricing it uses an even shorter window if there is enough data

#

for most currency type items it only uses listings within the last 2-4 hours since there is more than enough data

modern snow
#

Makes sense. I'm worried about low-volume areas, since it was my impression the listing index date doesn't get updated when an account logs back in. Maybe that won't be as much of a problem as I'm thinking

craggy beacon
#

it depends on what you are tracking really

#

if someone has had a divine listed for 2 days and it hasn't sold yet the price is probably not worth considering

#

for things with lower volumes you might need more than 48 hours of data but those tend to not be the type of items poestack/poeninja track

modern snow
#

One example that comes to mind is buy orders. Some folks frequently list a single, long-lived buy order that they don't modify for quite a while after it goes up. (Or I wouldn't expect them to modify it for a while)

craggy beacon
#

the other thing to note is you will get a stash tab update if any item in the tab is updated

#

so not only would that item need to have not changed in 2 days (note and quantity) but all other items in the tab would also be untouched

modern snow
#

Doesn't seem outside the realm of possibility, but either way it seems like an interesting thing to look into

craggy beacon
#

I guess if they list the div in a tab by itself and then don't change rate for 48 hours it would fall off

modern snow
#

Yeah, something like listing chaos for various essences in a tab and then not changing those rates for a while

craggy beacon
#

yeah but also not adding or removing any essence from the tab

modern snow
#

The note would be on chaos, not essences

craggy beacon
#

oh yeah buy orders

#

I don't really care about those

modern snow
#

shrugR seems like a niche enough thing to not worry about

craggy beacon
#

yeah I don't really use the buy order data for anything

#

it's tracked but unused

modern snow
#

the volume is also much lower, oftentimes it's odd to reason about

craggy beacon
#

yeah the data never felt stable enough that people should really be using it

modern snow
#

With the number of stashes you're tracking, how's the size of your stash data? I imagine for performance reasons, the calculated pricing data is historical, but the stash state is not?

craggy beacon
#

I keep a json file of the most recent listed version of each stash for use in some other features but this isn't stored in the DB so the size is fairly irrelevant, but for the pricing stuff I only keep track of the # of tracked items in each stash and what price they are at per account

#

That goes in the DB and snapshots are taken hourly to aggregate the prices and store them for historical reference

#

The DB size of public stash state is 4gb

modern snow
#

So you aren't actually storing the stashes themselves and all the item data therein, just the count of things you care about tracking?

craggy beacon
#

yeah the 48 hour thing is sort of for size but also just accuracy, I could keep a week easily but the prices get less accurate the longer time period you consider

craggy beacon
#

the whole stash goes into S3 so I can use it in other features

modern snow
#

Oof yeah for cost reasons I can't imagine myself actually making use of offsite object storage, etc.

craggy beacon
#

so far the stash data in DO version of S3 is only 50GB. DO charges $5 per month for 250GB of S3 storage so that might be the cheapest part of the site

modern snow
#

Ah neat, don't they also give you the first 5 GB free? Or is that not a thing anymore

#

Or "free" from the base $5/mo price

craggy beacon
#

If you needed historic stash data this would be way bigger but I only care about the last listed version

modern snow
#

Makes sense 😄

craggy beacon
#

I think you pay $5 per month to use spaces at all and that comes with 250GB

modern snow
#

Ah I see. Looks like their pricing changed

craggy beacon
#

then you pay like $.02 per gb after or something

modern snow
#

Zach do you happen to know if logging in re-indexes someone's stash tabs? I could test this, but figured I'd ask in case someone here knows

craggy beacon
#

I don't think it does

modern snow
#

Despair

#

Okay, thanks for your insights

velvet fog
compact isle
#

the pubDate reflects the most recent edit

velvet fog
#

got it, thanks

lean phoenix
#

Hi GGG introduced PKCE(Proof Key of Code Exchange) to OAuth flow I implemented this in my site to filter personalization it works perfect. You can try, check security, stability etc. If someone have question I can help with OAuth flow.
https://php10.5v.pl/simplefilter/

pliant cipher
#

So I’m writing a parser to pull out the translation data from stat_descriptions.txt, parser-combinators are good fun!

worthy cape
#

Heh, I was gonna say that they don't do Public Clients with PKCE yet, but the docs suggest that they support some of that now.
@compact isle This seems to be missing from the changelog 😉

civic crane
#

Novynn keeping it in secret to introduce with trade api 😄

compact isle
#

the swap-over to the new OAuth system wasn't quite as smooth as we'd hoped... so keeping it on the down-low until we're sure it's working correctly

worthy cape
#

😶

lean phoenix
#

But as I think it is not bad. For me took less than 1 h to adopt and change all the code. During this I improved session flow also and my classes. It was nice ;)....One You should add is fine session lifetime countdown during Authorisation login. Strange that it couldn't be done with standard HTML code, just in metadata tags. There is security connection icon, SSL, etc but not clear information about session...

light tide
#

anyone else having trouble getting an access token for a user today?

lean phoenix
#

Checked on my site works well...

light tide
# lean phoenix Checked on my site works well...

hmm ok thanks
Im getting 1: Error 403
this is whats in the header

2023-03-10 01:52:49 Date Fri, 10 Mar 2023 01:52:49 GMT
2023-03-10 01:52:49 Content-Type text/html; charset=UTF-8
2023-03-10 01:52:49 Transfer-Encoding chunked
2023-03-10 01:52:49 Connection close
2023-03-10 01:52:49 Permissions-Policy accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
2023-03-10 01:52:49 Referrer-Policy same-origin
2023-03-10 01:52:49 X-Frame-Options SAMEORIGIN
2023-03-10 01:52:49 Cache-Control private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
2023-03-10 01:52:49 Expires Thu, 01 Jan 1970 00:00:01 GMT
2023-03-10 01:52:49 Vary Accept-Encoding
2023-03-10 01:52:49 Server cloudflare
2023-03-10 01:52:49 CF-RAY 7a57eba82a376f9b-IAD
2023-03-10 01:52:49 Content-Encoding gzip

i think theyre rejecting my request maybe?

lean phoenix
#

Some HTTP status code ??

light tide
#

just starting working again... so strange

lean phoenix
#

Hymm This is during geting access token with CODE token ??

#

Did you have code token first ??

light tide
#

i got error just using the consent flow to get a token

lean phoenix
#

As I read this happens when is too many invalid requests in a short period, hard to help, You must check all code of program if it make correct requests, that is all what I can say ;/

light tide
#

ya all good, just strange cause i havent changed anything and suddenly stopped working

lean phoenix
#

I received this on updates in short times etc but good requests too, hard to say what is problem....

light tide
lean phoenix
#

Public Clients

#

As I see yes, but I have no experience with local app :/

light tide
#

ty for the chat

formal kraken
#

Is oauth broken atm or do I need to make some changes after the recent update?

image.png
#

Weird, filterblade seems to be working, but it has basically the same uri parameters, just asks for more scopes

craggy beacon
#

I got a 500 Unknown Error after the redirect back

formal kraken
#

almost looks like because the token already expired chaosthinking

modern snow
#

I was able to log in just now

craggy beacon
#

sometimes I had that happen when I accidentally consumed the token twice 🤔

modern snow
#

After clearing my cookies and trying to log in again, I get the 500

formal kraken
modern snow
#

Both. I cleared both, logged in on pathofexile.com, then tried your login flow

formal kraken
formal kraken
#

mh, maybe I get rate limited. Thanks for the help, have to add a bit more logging

lean phoenix
#

In my site works fine. Couple days agoo I must implement PKCE for token grand flow. Now works great🙂

pliant cipher
#

ooooo pkce support! Desktop devs will be very happy

lean phoenix
#

That is why some of them have disconnection every 10h, They must implement this, it is no so hard. For me took 1h to change all codes and btw upgrades main php class too...

trail depot
#

Does POB's Trader let you search by something other than DPS? eHP for instance?

oak estuary
pliant cipher
#

Wait... im writing my parser for the stat_translations.txt file for a project of mine

#

and my parser suddenly failed on a specific translation

# 
description
    1 minion_maximum_mana_+%
    2
        1|# ""Minions have {0}% increased maximum Mana""
        #|-1 ""Minions have {0}% reduced maximum Mana"" negate 1
    lang ""Portuguese""
    2
here >> 1|1|# ""Lacaios têm {0}% de Mana máxima aumentada""
        #|-1 ""Lacaios têm {0}% de Mana máxima reduzida"" negate 1
    lang ""Traditional Chinese""
    2
        1|# ""召喚物增加 {0}% 最大魔力""
        #|-1 ""召喚物減少 {0}% 最大魔力"" negate 1
#

I think I found a bug in the official files, "1|1|#" is invalid, it's the only place a format like that appears and it's different from the rest of the translations

worthy cape
#

Artifacts in human-editable file formats? Unpossible 😉

pliant cipher
#

Who's the poor soul who has to go in and edit that LOL

worthy cape
#

I've seen approximately all the typos in the various formats of PoE over the years, this one is new to me.

pliant cipher
#

Just thought it was funny

#

yeah my parser was failing and im like wtf is this

worthy cape
#

You should see the ones in terrain and actor files, the more common it is that it's hand-edited the worse it gets.

pliant cipher
#

it might have been patched cause that file is at least a week old

#

I bet hah

worthy cape
#

Anything that the game parser accepts goes, and probably some more that doesn't.

pliant cipher
#

Now I gotta account for errors... oh boy lol

worthy cape
#

Same error in the current version.

pliant cipher
#

it's probably been a while

#

I wonder what the text says in game

worthy cape
#

Stat not present at all in 2.6.3, error in 3.19.2b.
I need more extracted copies...

pliant cipher
#

I love this library im using to parse this stuff but it hurts my brain when I have to do anything more complicated than parse a few sequences in a row... idk how in the hell i'm going to do this now LOL

worthy cape
#

I'm happy that I do most of my parsing with regexes and string splitting in poe-cpp. I know from experience that strict parsers tend to be hard to adapt when it comes to such aberrations.

pliant cipher
#

im using a library in C# called Pidgin, it's a parser-combinator, so you kinda define it more functionally

# 
    public static Parser<char, TranslationSection> _translationSection => Tok(
        from lang in Tok("lang ").Then(_string).Optional().Between(SkipWhitespaces)
        from lineCount in Tok(Int(10))
        from lines in _translationLine.Many()
        select new TranslationSection(lang.HasValue ? lang.Value : "English", lines.ToList()));

    public static Parser<char, TranslationLine> _translationLine => Tok(
        from values in _statValue.AtLeastOnce()
        from text in _string.Before(Any.SkipUntil(EndOfLine))
        select new TranslationLine(values.ToList(), text));
#

looks complicated but you kinda build it up bit by bit

#

the "from" syntax is read like a script

#

Tok just means token, so anything before a whitespace

#

so a translation line is a string with at least one stat value, and "string text like this", then skip til the end of the line.

#

p neat imo

#

has made my life a LOT easier trying to write this

worthy cape
#

It's like that in 3.11.2b too, so probably forever.

pliant cipher
#

F

worthy cape
#

This was even before they changed the formatting syntax, heh:

1|1|# "Lacaios têm %1%%% de Mana máxima aumentada"
#|-1 "Lacaios têm %1%%% de Mana máxima reduzida" negate 1
pliant cipher
#

now THATS a syntax

worthy cape
#

I forget how many leagues ago it was they changed to more fmt-like placeholders.

polar hornet
#

Hi. I am trying to get a list of all active skill gems from .dat64 files. Is there a way to filter out gems like https://poedb.tw/us/Icefire (idx 3 in the table SkillGems.dat64) that cannot be used by the player?

poedb
Icefire
formal kraken
#

This is strange, it almost looks like there is a race condition after exchanging the token and when it becomes actually usable. I can exchange a token fine, but the profile lookup (https://api.pathofexile.com/profile) sometimes fails with a The access token provided is invalid or has expired. This happens less often in my local env than on 'production', which makes me lean even more towards a racy-thing 🤔

civic crane
polar hornet
odd aspen
#

I'm trying to find the image files for the item frame art for various item types/rarities (like the included image, but without the text on it). I would have expected to be able to find it among game files, but I've used VisualGGPK2 and looked pretty thoroughly and haven't been able to find them. Is there anywhere else I should consider looking?

image.png
worthy cape
#

It's made up of virtual images in sprite sheets, mapped by Art/UIImages1.txt, see the previous discussion: #tooldev-general message

#

Note that the current used art is the non-4K one, supposedly.

odd aspen
#

Fantastic. This is exactly what I was looking for. Thank you!

#

Think I had overlooked it because the sprite sheet was getting clipped due to screen dimensions

worthy cape
#

The current foil effects are computed live via vertex shader, but this ought to have all the traditional rarities.

formal kraken
# 
        .select(|file: &File| file.id.starts_with("Art/2DArt/UIImages/InGame/ItemsHeader"))
        .select(|file: &File| {
            file.id
                .starts_with("Art/2DArt/UIImages/InGame/ItemsSeparator")
        })
        .select(|file: &File| {
            file.id.starts_with("Art/2DArt/UIImages/InGame/") && file.id.ends_with("ItemSymbol")
        })

Should be all you need, as zao mentioned from UIImages1.txt

pliant cipher
#

Yay, my parser works!

#

I spent like 2 hours yesterday trying to work around the problem line but... I figured it's easier to just fix it manually lol

#

turns out it was

frank drift
formal kraken
frank drift
formal kraken
formal kraken
velvet fog
#

yes it's hard to maintain

worthy cape
#

I had some fun problems this league with the new shader stage Animation and extension blocks using generate in actual materials I cared about, I don't quite understand the intended semantics.

trail depot
#

Has anyone considered using machine learning with PoB? Curious if there are any repos out there for it.

worthy cape
#

Many have mused about it, not sure if anyone actually managed to build something.

formal kraken
oak estuary
hazy fiber
real vault
#

Cgrffh

#

Gtddchi

light tide
#

hi guys, im trying to handle retrys in my code

2023-03-16 04:29:41 X-Rate-Limit-Account is in the dict/headers max hits: 2
2023-03-16 04:29:41 X-Rate-Limit-Account is in the dict/headers period: 10
2023-03-16 04:29:41 X-Rate-Limit-Account-State is in the dict/headers current_hits: 1
2023-03-16 04:29:41 X-Rate-Limit-Account-State is in the dict/headers period: 10
2023-03-16 04:29:41 X-Rate-Limit-Account-State is in the dict/headers active time: 0,4

im waiting whatever the period is specified as and also not hitting the api if current = max but im still getting 429s on the 5th or so call..

was thinking I need to use the active time field somehow? has anyone used this?

modern snow
#

If the third number in the triplet from X-Rate-Limit-Account-State is not 0, you are currently rate-limited and need to wait at least that many seconds. If that's what you mean by active time field, then yes, you need to be looking at that and waiting that long before issuing another request.

light tide
#

wait you said makes 100% sense to me though

#

like i make a call at this timetamp

2023-03-16 05:17:10 X-Rate-Limit-Account-State is in the dict/headers active time: 0,5
... next call..
2023-03-16 05:17:23 X-Rate-Limit-Account-State is in the dict/headers active time: 0,6

active time still not zero and ive waited more then the period? that doesnt make sense to me haha

modern snow
#

Which policy and endpoint is this?

light tide
#

./character

light tide
#

hi guys, sorry to keep asking about this but im really stuck and i dont want to get 429s

2023-03-16 21:53:33 interval time check - intial call for period and retry time
2023-03-16 21:53:33 X-Rate-Limit-Account is in the dict/headers max hits: 2
2023-03-16 21:53:33 X-Rate-Limit-Account is in the dict/headers period: 10
2023-03-16 21:53:34 X-Rate-Limit-Account-State is in the dict/headers current_hits: 1
2023-03-16 21:53:34 X-Rate-Limit-Account-State is in the dict/headers period: 10
2023-03-16 21:53:34 X-Rate-Limit-Account-State is in the dict/headers active time: 0,2
2023-03-16 21:53:34 current_wait_time_extra 12
... 
2023-03-16 21:54:55 interval time check - intial call for period and retry time
2023-03-16 21:54:56 X-Rate-Limit-Account is in the dict/headers max hits: 2
2023-03-16 21:54:56 X-Rate-Limit-Account is in the dict/headers period: 10
2023-03-16 21:54:56 X-Rate-Limit-Account-State is in the dict/headers current_hits: 1
2023-03-16 21:54:56 X-Rate-Limit-Account-State is in the dict/headers period: 10
2023-03-16 21:54:56 X-Rate-Limit-Account-State is in the dict/headers active time: 0,6
2023-03-16 21:54:56 current_wait_time_extra 16
2023-03-16 21:54:56 retry after 300 for user xyzxcyz

i increase the time between calls as the active time increases + the period and i still get a 429 after 5 or so calls. I really dont know why.

civic crane
#

@light tide do you handle all rules, or only the first one. asking because you seem to split only by colon and have "active time: 0,2"

light tide
#

i dunno what the 0 is for haha

civic crane
#

@light tide

image.png
worthy cape
#

Consider something like this:

x-rate-limit-account: 45:60:60,240:240:900
x-rate-limit-account-state: 1:60:0,3:240:0

There are two distinct rules here for the account rule.
One where you get 45 requests over 60 seconds and if you violate it, you get 60 seconds of cooloff.
One where you get 240 requests over 240 seconds and get 900 seconds of cooloff.

light tide
#

oh jesus, i didnt know that

#

ok that makes alot more sense

#

imma ask chat gtp to give the functio to handle this hahaha

worthy cape
light tide
light tide
#

hi guys, does anyone have an example on how to handle all the rate limits and the different timeouts if you exceed? any help would be great! a link to a github is enough, thanks for your help

civic crane
light tide
#

ty ill take a look, im suprised i cant find any xrate limit examples in general online

worthy cape
#

It's been ages since I wrote mine but I think my approach was to keep a queue of past request times and shift them out as the lose influence. They govern when the next request can happen and in case of a rate limit event (due to other programs on the account/IP also making requests) I honor the state.

hexed mesa
#

also a reminder these rate limits are dynamic

modern snow
#

Mine is also similar to what zao described: keep past request times to understand the state and be able to apply the rule as seen in the latest response, with some added jitter to account for timing issues. Handling the case of an actual timeout is easy if analyzing the state in the response tells me I've actually crossed a limit.

rigid oxide
#

does anyone happen to know if skill activation points are stored in ggpk somewhere?
e.g. for dominus slam, he does a chargeup, then there's an explosion, and then he does the backswing animation, i'm interested in knowing where in the skill animation the damaging explosion actually happens

worthy cape
# rigid oxide does anyone happen to know if skill activation points are stored in ggpk somewhe...

I haven't looked at them in particular but I believe it might be in the .ao file associated with the animation.
I did a deep delve into the Elder's pirouette skill a while ago with the PoB gang:

    animation = "doom_spin"
        1.2 = "contact 5.66995 -51.4024 -209.997"
        1.2 = "contact 71.7129 -10.1348 -432.326"
        1.225 = "contact -10.4145 -83.4963 -368.355"
        1.225 = "contact 18.1417 86.7646 -371.073"
// ...
        1.975 = "contact -23.54 76.3816 -373.714"
        1.975 = "contact 49.3939 -65.5667 -379.359"
        3.05358 = "hide 0 0 0"
        3.15004 = "teleport 0 0 0"
        3.30148 = "reveal 0 0 0"
        4.5 = "slam 0 0 0"
#

Those contact fire hardcoded mortars while the slam, well, is a slam 😄

#

There's some additional flags elsewhere that refine the meaning of the coordinates, like if contact-is-a-normal, contact-is-an-offset, etc.

rigid oxide
mint basin
#

Is there any ToS friendly way to query which quests a character has completed?

grave wren
#

iirc only reading chat log

#

so nothing like getting a global quest status somewhere

mint basin
#

Okay, thanks! Will have to investigate that even further then.

mint basin
#

unlucky, doesn't seem like anything related to quest completions is printed there 🥲

worthy cape
#

Seems about right, the information story there is bad but at least everyone are equally hosed 😄

trail depot
#

How fast will PoB be updated for Krangled, if at all?

worthy cape
#

Is that one of the events? I don't think there's any plans to assist in that.

trail depot
#

Yeah, starts tomorrow. Randomizes the passive tree

worthy cape
#

Way too short lead time and none of the data available that PoB uses.

trail depot
#

Figured that was the case but thought I'd ask. Wasn't sure how big a lift it would be or if GGG provides the passive tree in a consumable format for PoB or not.

worthy cape
#

A bit of the point of the league is to have organic human interaction in sharing revelations and strategies around the tree, it would kind of defeat that intent to spoil it.

#

Normally we rely on and postprocess the JSON files published shortly before league start in conjunction with some game files.
Other people use .PSG files from the game for the tree representation and go from there.

#

The mechanism with which the event tree is computed is unknown to everyone and may intentionally be quite opaque and non-static.

#

You need both the data, the effort to figure out how it's transformed, and most importantly, buy-in from tools people to do the work.

velvet fog
#

there is no new .psg, and there is no new field in PassiveSkills, so there is a high probability that it is hard-coded in the executable file, in order to avoid being datamined

hazy fiber
#

its possible its also a seeded algorithmic scramble, with the event setting the seed itself

stoic locust
#

They made a fog of war thing for the passive tree with the intention of hiding the whole tree except for the nodes next to active nodes... I'd imagine they have no intention of it being [easily] viewable in its entirety.

formal kraken
#

I still hope that every private league gets its own seed

stoic locust
#

And the main thing about the event is it being randomized and hidden from view, so idk... seems counterintuitive to the league itself to want to be able to preview the entire tree before playing it

hazy fiber
#

so its not unreasonable that the whole tree will be viewable with a fixed seed within 5-6 hours of league launch given a few volunteers and a tooldev willing to do it, without even needing datamining

#

but I assume no one is gonna go to the effort

#

which is good

formal kraken
#

yeah it's just gonna be one different tree which is known after a few hours, then the build makers start making builds and the rest is gonna copy

#

would be cool if everyone could just have their own version and/or maybe even fish for a good tree

frank drift
#

that would be cool but I can't see that level of effort for a 1 week event

hazy fiber
#

also on a different note, is ggg tracker broken, its only been the twitter posts for a while not the forum posts?

#

last non-twitter post in that channel was match 2nd?

worthy cape
#

Imagine all the people complaining about having bad seeds on their account or characters. Rerolls would be a blast. brittlPopcorn

velvet fog
#

same tree confirmed

trail depot
#

I like importing my existing character into PoB and using it to find powerful nodes. I can get a general sense of how powerful a node is, but sometimes it suggests things I hadn't considered. Or that are counter-intuitive but effect.

compact isle
#

try and keep non-tool-dev related things out of this channel please

robust sentinel
#

guys, does anyone know how I can contact the developer of poe.trade? I have a question about the website, but couldn't find any contact info

formal kraken
worthy cape
#

All of xyz's PoE-related services are defunct now, I believe.

robust sentinel
worthy cape
#

poe.trade, I think poe.party was theirs too.

worthy cape
#

I don't know if they've got any discord presence anymore, the name is great for blending in 😉

robust sentinel
frank drift
#

they were here at one point, may or may not be still

pallid wraith
#

There's one xyz guy here with thousands of messages sent

#

And the tool dev role

oak estuary
#

@chrome topaz is the one

worldly adder
#

Hey all, I'm trying to get my OAuth2 working and I need help troubleshooting the 403 Forbidden error I get during the Access Token request step.

Is here a good place to discuss in full detail?

formal kraken
worthy cape
#

X-OC-Do-Not-Steal: hunter2

worldly adder
#

I've managed to get the user permission steps to work just fine in both GoLang and Python. I get so far as to landing back on my URI where the code and the state are in the URL but when attempting to request an access token with a POST https://www.pathofexile.com/oauth/token to I get the 403 forbidden.

I'm making sure to send the request with a header Content-Type: application/x-www-form-urlencoded and then including the ID,Secret,grant,code,uri,scope in the body

formal kraken
worldly adder
#

I'm not recalling that term, let me read up. Likely not.

formal kraken
worldly adder
#

Okay so if I'm reading this right there is a second header User-Agent: SOMETHING that I'm likely missing. Do you know where I can find documentation on it? I didn't find anything mentioning it when I read here https://www.pathofexile.com/developer/docs/authorization

formal kraken
worldly adder
#

I see you have yours just set const POE_API_USER_AGENT: &str = "OAuth pobbin/1.0 (contact: ggg@pobb.in)"; I should be able to just copy that formatting?

formal kraken
worldly adder
#

thanks!

hidden forge
#

Where does poe.ninja gets its data from?

#

is there an api for the trade website?

formal kraken
hidden forge
#

items

frank drift
formal kraken
#

the stash tab river

hidden forge
#

thx

formal kraken
#

I think you need api credentials now to access the river

hidden forge
#

yeah

frank drift
#

indeed

hidden forge
#

im not interested in accessing it , was just wondering how the bots that whisper me get their info

frank drift
#

bots are unlikely to be using the river, more likely to be using trade site alerts (pretending to be a web browser)

#

unless you don't mean flipping

hidden forge
#

no, not the flippers

#

the ones that try to snipe stuff

craggy beacon
#

the river has a 5 min delay

frank drift
#

that's flipping yeah, the river is useless for that

hidden forge
#

arent the flippers the one that just sit in h/o and act like change buro?

frank drift
#

depends on your definition, but generally it means buying underpriced items and reselling for a higher price

worthy cape
#

They're more likely to use the trade site and abuse live search sockets.

hidden forge
#

I see

craggy beacon
#

I feel like almost all snipers flip items, but not all people flipping snipe items

hidden forge
#

wait, sockets? how do they hook up?

#

dont you need the source code for that?

worthy cape
#

I mean, getting live results by using the websocket machinery and then doing naughty things to the client to automate snipes.

hidden forge
#

yeah, but how do they access a websites socket

worthy cape
#

In any way, largely uninteresting as it's nothing one should do.

hidden forge
#

so its not like

#

official tech?

worthy cape
#

"websockets" is a particular technology that lets a webside open up a bidirectional frame-based communication channel to the server.

#

The trade site uses it to notify the page of new items on live searches, which the page then requests the details of.

frank drift
#

websockets are how the live search functionality on the trade site functions, but they're not supposed to be used outside of that context

hidden forge
#

oh so they open their own socket and hook to the server?

#

isnt there any way to prevent that ?

craggy beacon
#

not easily since they can just look at how your browser is doing it

worthy cape
#

Not really, web tech requires the code to run on the user's computer in a program that the user partially controls.

hidden forge
#

I see ,thx

frank drift
#

security is always a cat-and-mouse game

mint karma
#

Finally got some time to work on the expedition regex page, I will work on the UI and try to make it a bit more user friendly before next league but the core features are there! And just in time before xanthics stops updating his version!

Any feedback / bug reports are more than welcome https://poe.re/#/expedition

Path of Exile Regex

Path of Exile Regex

civic crane
frank drift
#

fun police

craggy beacon
#

It seemed like a bit much but I like rabbitmq so didn't say anything KEKW

outer jay
#

I'm toying around with querying character data for our private league and I'm a bit confused by the rate limits, since I'm querying characters I should only be worried about IP and Account rate limits right? Is this a fixed or sliding window?

worldly adder
#

I'm getting still getting 500 ERR response when attempting my Auth token request. I see the docs suggest to try again later. I've given it 12 hours. Anyone know how long I should expect to give it before trying again? If I change content in my request body/data I get a 400 error indicating something is wrong so I have reason to believe I have the correct pieces to my request but if anyone has suggestions or knows how long to wait that would be nice.

My request looks like this:

Method:  POST
url:     https://www.pathofexile.com/oauth/token
head:    {'Content-Type': 'application/x-www-form-urlencoded', 
          'User-Agent': 'OAuth chipytools/0.0.1 (Contact: contact@chipy.dev)'}
data:    {'client_id': 'chipytools', 
          'client_secret': 'HIDDEN', 
          'grant_type': 'authorization_code', 
          'code': 'HIDDEN', 
          'redirect_uri': 'https://chipy.dev/poe_auth.html', 
          'scope': 'account:stashes account:profile'}

<Response [500]>```
worthy cape
#

Shouldn't there be a code_verifier as per the dev docs?

formal kraken
worthy cape
#

That's the only place I see authorization_code mentioned.

formal kraken
#

ah right, that looks like the issue

#

mixing PKCE with the client credentials grant

outer jay
worldly adder
#

Okay well I did try add in my Verifier but then I get a 400 error, so maybe I have the wrong encoding for my verifier?

code_verifier = $base64url_encode($secret)
code_challenge = $base64url_encode(sha256hash($code_verifier))
worldly adder
outer jay
#

What is your request url?

worldly adder
#

Hang on... lets me get some samples

#

https://www.pathofexile.com/oauth/authorize?client_id=chipytools&response_type=code&scope=account:stashes%20account:profile&state=HIDDEN&redirect_uri=https://chipy.dev/poe_auth.html&code_challenge=HIDDEN==&code_challenge_method=S256

*Looks like I didn't type out my URI right above https://chipy.dev/poe_auth.html is the correct value as it's only stored in one variable for my entire script

#

Then after that approval I get: https://chipy.dev/poe_auth.html?code=HIDDEN&state=HIDDEN

outer jay
#

The redirect_uri in your token request has to match exactly I found

worldly adder
#

Okay I'm going to rebuild a fresh sample test

# 
            'client_secret': 'Fav8k1XoxIaj', 
            'grant_type': 'authorization_code', 
            'code': 'HIDDEN', 
            'redirect_uri': 'https://chipy.dev/poe_auth.html', 
            'scope': 'account:stashes account:profile', 
            'code_verifier': 'HIDDEN'}```

this gives me a 400 error

```data:    {'client_id': 'chipytools', 
            'client_secret': 'Fav8k1XoxIaj', 
            'grant_type': 'authorization_code', 
            'code': 'HIDDEN', 
            'redirect_uri': 'https://chipy.dev/poe_auth.html', 
            'scope': 'account:stashes account:profile'}```
This is the 500 error again

I made sure that the URI is all referenced back to a single place so I believe that if the user permission request completes I believe that means the URI matched what PoE has on file
outer jay
#

Your secret is in there

worldly adder
#

hahah RIP well i guess I better generate a new one :p

outer jay
#

I pass the scopes directly from the query so that there's no difference

#

I do uri encode it before hand though on the website

worldly adder
#

mmmm

outer jay
#

Also, I'm guessing in the second example, you're removing the code challenge from your url right?

worldly adder
#

Oh... no I'm not putting any of it in the URL I'm doing a POST request to https://www.pathofexile.com/oauth/token with that "data" encoded with x-www-form-urlencoded formatting in the payload

#

is that wrong? Should I have it in the URL too?

outer jay
#

I mean the authorize url

worldly adder
#

oh no, I didn't remove. I'll try that quick

#

That might have done it? So the auth request doesn't use the Code Challenge anymore?

#

Heck yeah I got an ACCESSTOKEN!

#

❤️ @outer jay

outer jay
#

Nice

#

That bit is only for PKCE so not needed

worldly adder
#

ahhhhh

#

TIL

modern snow
mint karma
mint karma
modern snow
#

Yeah, precisely

mint karma
#

any ideas about what a sane default value for that should be?

modern snow
#

I think what you have now is fine. The T0 belts, all the way down to Divinarius-ish

#

My friend and I exclude Imperial Skean and Sage Robe, but that's just us

frank drift
#

"most expensive N per currency" would be most useful IMO, not sure if that's what you're doing, pretty sure xanthics's did something like that

#

gwennen is a waste of time anyway post-nerf but I guess it's still fun

modern snow
#

I love random gambling stuff like that, so I often do it with spare expedition currency haha

mint karma
#

gwennen is also pretty nice in private leagues / SSF if you dont aim for the T0-T1 bases!

mint karma
frank drift
#

fair

modern snow
#

@mint karma Maybe it should change based on the selected economy?

mint karma
#

Good point, hmh, I think I will leave it blank or 0 though. So people can add that filter if they want to but it will be off by default

modern snow
#

@civic crane I was looking into making a PR on APT to allow bulk searching non-unique maps. I think that outside of the first couple days of league, many people who are looking to sell their maps are interested in bulk data.

However, it looks the item data njson files need to have the trade tag/note code in the data to make this happen more easily. It seems the scripts for generating it are not in the repo. What do you think about including the maps at different tiers with their note codes to allow them to be bulk searched?

civic crane
#

couple of issues with them, better create issue)

modern snow
#

Sure, I'll do that

modern snow
#

Created #898

sudden ocean
#

Whatup, anyone around? I am in the process of learning how all of this works and I am using postman to interact with the API. I have tried generating my token using the auorization section in postman and when I click the orange button shown here I get a message that doesn't go away. saying it is collecting a token. This goes on until I hit cancel. If I check the console I can see that the token request was sent and received and a token was issued but it is not getting stored automatically the way postman seems to try to store it. Anyone familiar with postman know what I am doing wrong? Should I just give up, put in my token details manually and leave it?

image.png image.png
sudden ocean
#

I never figured it out. I ended up just saving the token manually.

modern snow
#

Not sure who to let know about this, but the official API docs section on error codes could be improved to include 403 Forbidden and note that if you try the wrong method on some endpoints (like POST to /leagues), you'll get insufficient_scope back. Or better yet, return a 405 Method Not Allowed (if that method is not supported at all).

formal kraken
#

anyone remember what the ascendancy images are called in the game files? @worthy cape maybe?

Elementalist.png
#

Seems to be Art/2DArt/UIImages/Common/IconDexInt_Assassin e.g.

worthy cape
#

Probably. You could likely see filenames on the wiki articles on the classes.

mint karma
sudden ocean
#

Am I allowed to send gets to https://www.pathofexile.com/api/trade/data/items? Sometimes it works fine and I get an expected response and sometimes I get error messages saying "Bearer realm="pathofexile:production", error="insufficient_scope", error_description="The request requires higher privileges than provided""

modern snow
#

Does anyone have a public stash change id for about the start of Kalandra?

compact isle
#

1710040184-1712246230-1654997017-1834736995-1779607706 is what I get

modern snow
#

Thanks

compact isle
sudden ocean
#

Oh. That's really helpful. Thank you so much!

#

I actually expected the opposite to be true. Could you help me understand why things like api/leagues exist separately from /league (which requires oauth) and why league needs oauth to get and leagues doesn't?

#

Let me get the links actually so I'm being clear.

compact isle
#

/api/leagues is an internal website API that we use to power the ladders on the site. Essentially anything outside of the documentation is considered internal and is not intended for third-party use

sudden ocean
#

Oh I see. That was what I was wondering because the trade search features are behind that internal website API.

compact isle
#

indeed, they are not intended for third-party usage. We don't offer any support or encouragement in their use since technically it's against our Terms of Service

sudden ocean
#

thanks for clearing that up for me 🙂

sudden ocean
#

now i havent sent any actual requests to it yet but im a little confused about the public stash api. if the results are limited at 255 stashes how do you get them all? how does the pagination thing work? i dont really understand it.

#

does anyone have maybe an example of it in action?

modern snow
#

The developer docs, which I assume you've read, give a pretty good overview. The change ID you're giving to it marks certain stash changes at some point in time. Their servers will try to pack a bunch of stash changes newer than those, up until some maximum limit. (They say 255 stashes, but I've seen responses with more than that sometimes.) The response will also have the next pagination code to use to get the stashes after the ones that got returned to you.

#

Over time, you'll catch up to the latest publicly available stash changes. From there, just keep polling the endpoint to stay caught up to reality (minus 5 minutes).

#

As a grossly simplified example, say you requested public stashes and gave it change ID 1-1-1-1-1. You might receive about 255 stash changes, each with an ID greater than 1. But the server has to stop somewhere, so it responds with a next_change_id of 300-301-302-303-304. You'll take that and use it in the next request to the endpoint to continue getting more stash tabs, leaving off where the last request stopped.

sudden ocean
#

Okay so when you send the request without an ID initially.. are those the most recent stashes? Where does it begin?

worthy cape
#

The start of the river is with all shards of the ID at 0 or so. Not sure where a request without ID is but you can probably guess based on the next ID it reports.
Ninja has a "current ID" value on a stats page somewhere too.

sudden ocean
#

ohhhhh okay

worthy cape
#

Note that older versions of a particular tab will be empty if superseded by a newer version, so there's not that much point in going very far back.

#

You can technically fetch everything from the very beginning of the river but its going to be full of such holes.

#

I think you can detect that you've caught up for a moment when the next ID is what you put in.

sudden ocean
#

thank you so much. so the highest shards have the most recent changes and it goes back from there but older pages are less relevant and more likely useless so it just chooses a start point somehow.

#

the docs says if youve caught up the array will be empty and repolling with the same next_change_id will show new results when theyre available

#

I definitely understand now between the explanations here and the docs

worthy cape
#

Say that I have a tab that with content changes at shard ID 4 and 6.
If you fetch 4 before 6 is published, you get the state of the tab as it were at 4.
When 6 is published fetching 4 again gets you a placeholder and fetching 6 gets you the new state at 6.

sudden ocean
#

it probably takes a snapshot every 5 minutes and publishes that. and then if more than 255 stashIDs change state it creates additional pages.. i guess.

worthy cape
#

I believe it's fairly tightly tied into how the game aggregates changes you do to your tabs in game, like how they're batched up either with a timeout or when you zone.

sudden ocean
#

i could be way off and i dont want to confuse myself since i think i have the general idea

worthy cape
#

(there's a reason why you always get whispers when you just entered a map 😄 )

#

In general, follow the river by chaining IDs and you'll get most of the update contents.
If you're curious about what happens to old data you could make up old IDs and look at what the data does.

sudden ocean
#

now that i have a token and i got postman running i need to learn how to store the data i fetch

worthy cape
#

Depends a bit on what your ultimate purpose is too.

#

Consuming the river is a great exercise in both pipelining requests and data ingestion and filtering/processing.

sudden ocean
#

my ultimate purpose is developing broad cs skills. my initial thought with poe was the manipulate poeninja data to create metrics i found useful but as i got into it i figured i may as well just go to the source and everytime i move an inch i see another mile.

#

which cool. im just enjoying the journey. im learning things ive wanted to learn for a long time.

velvet fog
worthy cape
#

Oh right, the official site has it.

modern snow
#

To your question about the beginning of the river, it's somewhere around 4000 last I checked. As far as I know, the river has pretty much all the changes since it began, so you could even start consuming from the beginning of Sanctum to paint a picture of all public stash tabs throughout the league.

formal kraken
modern snow
#

+1 on ES, and I believe it's also what GGG uses to power trade and psapi behind the scenes

outer jay
compact isle
#

frameType

Normal = 0
Magic = 1
Rare = 2
Unique = 3
Gem = 4
Currency = 5
DivinationCard = 6
Quest = 7
Prophecy = 8
Foil = 9
SupporterFoil = 10

outer jay
#

Ah cheers!

sudden ocean
copper steeple
copper steeple
sudden ocean
#

is elasticsearch free?

#

im seeing self managed basic free download but does that transition into me needing to make a purchase

copper steeple
sudden ocean
#

oh nvm just click the faq me

#

it is free. i used their website and they have a cloud product that isnt free

formal kraken
#

they changed the licensing "recently", but that doesnt matter for you

#

but if you're worried there is also OpenSearch (forked from ES)

copper steeple
#

True that's what it is called

#

I was looking for it because I saw it in github/trending a while ago but couldn't remember the name

sudden ocean
copper steeple
#

Yes

sudden ocean
#

okay im getting overwhelmed a bit here. opensearch is recommending i use docker to set it up.

so far i have just been using postman from my pc to mess with the api. im still very new to cs. should i pause here and take some time to learn about docker?

formal kraken
#

you don't have to use docker, but it certainly makes it easier. You can decide if you want to learn/look into docker, or just 'use' it without understanding it necessarily and continue from there

sudden ocean
#

when i say new i mean im doing harvards free cs50 course and im still on the project 0 in scratch

formal kraken
#

let me check, I should have a docker-compose ready to go for elasticsearch somewhere

sudden ocean
#

which btw.. scratch is awesome

#

my 400 year old mom is doing cs50 with me using scratch so far.

#

i guess the most important question moving forward is why does docker make it easier?

#

and as a student let me try to answer first based on a guess

#

it gives me a clean environment to start from

worthy cape
#

A good stab at an answer 🙂

sudden ocean
#

ty zao

worthy cape
#

The installation procedure is way easier when you have a known operating environment to install the software and its dependencies into, rather than support installation instructions and all the horrors that comes with targetting all sorts of distros and coexisting with other software and customizations.

sudden ocean
#

nice

formal kraken
worthy cape
#

A container has nice interface with the outside world in that it gets access to data via volumes/mounts/network and exposes some ports for the services it provides.

#

(they have their own suite of problems for which I'm running into approximately all the corner cases, but that's unrelated)

formal kraken
#

Well, it depends on your environment. I am just gonna assume you're on windows or macosx, then docker makes it a lot easier. If you're on a Linux, it is not necessarily easier.

sudden ocean
#

i am on windows

#

alright its docker time then. ty friends

formal kraken
#

oh god, there are actually no good resources on starting a single node elasticsearch docker compose locally

#

If I link you the docs you're gonna turn around and run far away

sudden ocean
#

im going to use opensearch

formal kraken
#

so unnecessarily complex

#

pretty much the same setup

#

opensearch was forked from elasticsearch, so a lot of stuff is the same

sudden ocean
#

ill just take it one step at a time. im reading about docker up front so that i can get it set up right when i use this link https://opensearch.org/downloads.html

OpenSearch
Opensearch 2.6.0

OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.

#

and then ill work on the postman integration and maybe hooking it all up to my github or something

#

ive got 3 projects going on atm so i have to stay focused. i mean dont you all end up with like a million ideas a day for things to do with your skills?

#

i learn one thing and it spawns a bunch of brain children

#

i want to use tortoise TTS to make a youtube channel that reads research papers like audiobooks i can fall asleep to and im working on the cs50 course and then this project. PLUS WE GOT A NEW LEAGUE SOON

formal kraken
sudden ocean
#

okay so save that in postman?

#

or just to my pc

#

yeah the offtopic thing. i get excited

worthy cape
#

If it's somewhat related to making tooling around PoE data and of interest to people following along, you could probably create a thread for it - should keep it nice and separated.

formal kraken
modern snow
#

By the way, there's no reason you couldn't set up a local instance of ES using Docker. Unless you have a compelling reason to choose OpenSearch as an alternative, you do have the option of just running ES.

worldly adder
#

I want my tool to look up a list of active leagues to easily select one (without worrying about selling) is the service:leagues scope the best way to do that? When I attempt to request it I get this 'error': ['invalid_request'], 'error_description': ['An unsupported scope was requested'] Which I'm guessing means that my authentication type or protocol is wrong ( I did verify that I'm registered for that scope with GGG)

modern snow
worldly adder
#

I believe I'm getting that error, when attempting that flow. I've been able to complete that authentication and receive a token before (to do my stash searching function) but I had GGG add the new scope and I'm troubleshooting the additional scope.

normally I'd put in the user-permission URL and it'd take me to GGG's authorization page, when I add this scope (separated with a space like my other scopes) it returns the above error instead of taking me to the approval page

#

This Failshttps://www.pathofexile.com/oauth/authorize?client_id=chipytools&response_type=code&scope=account:profile account:characters account:stashes account:item_filter service:leagues&state=dev&redirect_uri=https://chipy.dev/poe_auth.html
This Works https://www.pathofexile.com/oauth/authorize?client_id=chipytools&response_type=code&scope=account:profile account:characters account:stashes account:item_filter&state=dev&redirect_uri=https://chipy.dev/poe_auth.html

#

Should I be using the "PKCE" Code Grant method instead for service: type scopes?

outer jay
#

Client Credentials Grant

This grant can be used by an application in order to access services unrelated to an individual account that are exposed as part of the Path of Exile API. Unlike other tokens these ones do not have a set expiration time. They can still be revoked manually via your application settings page.

#

Look at that on the docs

modern snow
#

PKCE isn't necessary for service scopes. You're hitting the wrong endpoint to request service credentials. /oauth/authorize is for the authorization code grant. You want the endpoint for the client credentials grant, /oauth/token.

#

You don't need to navigate to GGG's website and log in for service credentials

worldly adder
#

ooohhhh let me see, so because it's the service scope i can omit it on this step and add it in when I do my code/token exchange?

#

oh nope, so I'd have to build two separate tokens, one for services and one for account?

modern snow
#

Like Navi mentioned:

This grant can be used by an application in order to access services unrelated to an individual account that are exposed as part of the Path of Exile API.
The leagues endpoint has nothing to do with individual accounts. There's no reason to go through the authorization grant flow when you can just get a set of service creds

outer jay
#

I just copy pasted from the docs, no need to credit me

worldly adder
#

I'm sorry but maybe my english just isn't understand what that means. They are saying since it's not private data there isn't need for OAuth... but I'm trying to translate that into programatic steps so this means I can skip the authentication step for non-private data requests

#

I'll need to look for examples on github to help it click in my brain

modern snow
worldly adder
#

Okay, will do. Appreciate your guiding me ❤️

modern snow
#

No problem

worldly adder
velvet fog
grave wren
#

Yep what I thought templarthumbsup

worthy cape
#

(note that RePoE is still without an active maintainer)

grave wren
#

I heard zao will happily maintain

worthy cape
#

Would consider to the detriment of personal health? Yeah.
Actually able to do any tooldev stuff at all lately? Nope.
As it looks now I won't be capable of doing anything for the next league in any project.

grave wren
#

Oh no 😦

sudden ocean
#

i got docker running. still need to spend some time learning it but i at least got it started and ran an image on it. been spending some time messing with some tts machine learning thing which has been tough but ive learned a lot. signed up for copilot too though i havent really any use for it yet

worthy cape
# grave wren Oh no 😦

In essence, I've been turbo-stressed about work, life and hobby dev for a long time while also having some related and unrelated physical and mental health problems.
Turns out that taking on every single problem in dev and being universally available to answer things people ask about PoE tooling isn't quite healthy.
I need to focus on myself for an indeterminate amount of time and get some changes implemented there.

#

I had hoped that someone would take up RePoE after 3.20 and provide a replacement for the data that many tools relied on but it doesn't seem like anyone stepped up.

grave wren
outer jay
oak estuary
golden knot
#

which also, related to the above convo about RePoE, seems like a good candidate for a new .dat source of truth

#

I'll very likely be trying to help out @civic crane in a big way with it in the future once I get rolling on a project I'm planning

hazy fiber
#

doesnt that just straight up use RePoE?

daring stag
formal kraken
#

I may add an exporter to my parser, but what worries is me if people start depending on the output and I actually have to maintain it ...

#

@daring stag btw was meaning to ask, does it help you if I add a CORS whitelist to pobb.in?

daring stag
formal kraken
#

Maybe if I add poe.ninja as well 🤔

daring stag
#

One less server for me to keep running soupSataniaLaugh but also don't want to shift maintenance responsibility around just more mental overhead for everyone involved

formal kraken
#

nah, this is basically free for me

#

@fickle yew Would you mind if I proxied your builds (poe.ninja/pob/<id>) through pobb.in/<id>?

velvet fog
frank drift
velvet fog
#

but I can't pass the old method right now

#

replied http code 400

formal kraken
#

Old method is still working for me

velvet fog
#

I got it, "error_description":"The redirect URI does not match"

fickle yew
compact isle
worthy cape
compact isle
#

was that really 3 weeks ago 😅

worthy cape
#

^_^

velvet fog
#

too busy these days

outer jay
#

I'm guessing you can always hit the IP rate limit using any API right? Haven't hit it when querying ~50 characters from different accounts yet so wondering if I should handle it properly

worthy cape
#

Does it mention the IP rule in X-Rate-Limit-Rules as contributing for those endpoints?

outer jay
#

You mean in the header?

#

Not currently, but since rate limits are dynamic, I'm not sure whether it'll show up at some point

worthy cape
#

Ah.

sudden ocean
#

i spent the last two days trying to get a github repo working. got it working and then found a better maintained version of roughly the same thing

#

so we begin again

#

maybe

inland kestrel
tough veldt
#

Hello there, quite "new" on making tool for poe (I've mainly done excel spreedsheet for my characters).
I've been working on something close to pobb.in/poe.ninja: a pobcode displayer in a web page using meta-is-beta/HoradricHelper to display. I now come to you for help, with a lots of questions!

I'm searching for site like brather1ng/RePoE to get acces to stats for the gems. I've find out that image are very hard to get by, but lvlvllvlvllvlvl/RePoE managed to get them.
Do you know of any other ways I could get images easily? (I mean, without extracting from the game files, just having a somewhat uptodate repo, but if not, see next question) I don't know if I can extract image with the good format from snosme/poe-dat-viewer.

I'm also french, so I'm trying to get the french translation of stat, I think there is no other way than to use OmegaK2/PyPoE (I've seen pogo and libggpk too) to extract myself the files and building my own repo. Could someone guide me what files I should edit to get this?

Also, I found out that using the stats files, I couldn't find out where are the description of the support gems (teal text), any idea?
And lastly, is there an api to get the passive skill tree data? I'm currently copy-pasting the json from the official site each league.
Optionnal: I've discussed quickly with the concepteur from th3conc3pt3ur/poe-roadmap, there is currently no generated file to have the list of available gem reward, gem vendor. I don't know if the wiki page is automatically done by chuansing's script or if it come from gamefile or if someone is uptdating it manually, but for poe-roadmap, it is done manually.

(If you dare look at my work in progress, it's at antilogos/SimpleBuild on github).
Thanks for anyone able to answer. Thanks very much to all the developpers I've based my code on!

hazy fiber
velvet fog
#

gem rewards added back, but gem vendor no.

#

most weapon/armor/unique can accessed by ggg's cdn

tough veldt
#

Nice! and not... I mean, that exactly what I was looking for, I've totally forgot that they had a github, but if I want the French translation, I would still have to extract it from game file or copy paste from official website. I didn't know it was in game file though.

hazy fiber
#

never actually reconstructed it from gamefile myself, but doubt its an easy extraction, yeah French translation would be harder to get as not on github

worthy cape
#

brather1ng's RePoE is indeed stale now, not sure where lvl's is at.
K2s' PyPoE is quite stale, the wiki has its own diverging PyPoE that handles dat64 and 3.20 but isn't suitable for external consumption, not sure again what lvl used.

#

Reusing an existing stat translation utility is a good idea as the format is a bit gnarly.

whole ruin
#

get GGG to host and maintain a simple API for storing/receiving pobs 😄

craggy beacon
#

headless pob isn't too hard to setup

frank drift
#

wonder how close to launch we'll get the skill tree

compact isle
#

today in theory

rocky lion
#

Lol good luck Novynn!

wind garden
#

lol for real

compact isle
#

lol I'm famous

worthy cape
#

Hi "one guy!"

compact isle
#

now taking suggestions

fickle tusk
#

take my energy @compact isle

civic crane
#

one guy sadge

velvet fog
#

.datl file removed, now .dat64 and .datl64 exists

inland kestrel
#

take my energy @compact isle lol

grave wren
#

is this the time were we ship some coffee packs to you

craggy beacon
#

Wonder if they could release a few example item JSON's with trees so tools could start updating

compact wigeon
#

Second the request for an example of how items will look in the API and/or ctrl+c

compact isle
#

no tree information, just the allocated node's mod text

compact wigeon
#

That's very helpful actually, means we just need to support the mod text to make it work. Cheers!

compact isle
#

I'll get some examples hopefully today

frank drift
#

will be interested to see what info filters get

compact isle
#

items don't drop with trees so no filter things turns out this is wrong, we're adding a function

odd sail
#

I assume the weapon tree nodes are tiered by value/rarity... if the mod text is organized by tiers it would be awesome

compact isle
#

the advanced text has tiers

odd sail
#

In Patch Notes:

Added a new Dexterity/Intelligence Support Gem - Manaforged Arrows: Supports bow attack skills, causing them to trigger when you've spent and amount of total mana on other bow attacks based on the supported skill's mana cost. Supported skills deal more damage based on their mana cost. Cannot modify the skills of minions.

that and should probably be a certain (... spent and amount ---> spent a certain amount)

compact isle
compact isle
formal kraken
#

I was gonna go to bed, but might as well update everything now

hazy fiber
#

zao already posted it elsewhere

formal kraken
#

okay, all projects updated, can go to bed now

dull roost
#

Hmm, can you no longer rename the data.json to tree.json and have pob create an lua from it?

hazy fiber
#

it does it from data.json not tree.json

#

but also theres already a PR up with the new tree

formal kraken
dull roost
compact isle
normal parrot
#

Meaning that if I list an item which has its crucible tree revealed, but nothing allocated will not show any information about the nodes and their connectivity on trade?

hazy fiber
#

trade is separate from item copy/paste info, but a decent question