tooldev-general

[simple ravine]

ya, they dont decouple the UI's main thread with tasks, so the UI gets laggy

mutters something about juniors

[frank kayak]

still have a crt tv here - collects dust though as i have no reason for tv

it just has to work somehow... most of the times then its sellable

most important: it has to look good

doesnt have to enable users to do stuff

at least thats the feeling im getting

[simple ravine]

gf watches some tv, i do it quite rarely

sometimes news or news-like programs

but it's more and more on-demand kind of entertainment for sure

[frank kayak]

i have YT - and sometimes i buy a month netflix

i dont need 7min ads in my scifi series

[simple ravine]

ya, netflix and disney+ for the kid, seedbox with 7tb storage and youtube

110011, u're german right?

[frank kayak]

ja

[simple ravine]

I have to rant.

...

SAP.

[frank kayak]

go on

[simple ravine]

can we burn that company to the ground please.

I'll provide the gasoline.

[frank kayak]

personally i might agree

but my cousin works indirectly for them i think

so not for now

[simple ravine]

I have some clients that are so entrenched

it's so sad.

their policy is "If we can get the <use case> to fit within SAP ecosystem, that's our first choice"

A monolithic piece of dumpster fire.

then when I talk about cloud computing, they're like "what about vendor-lockin?"

😂 🔫

then we need to integrate with other things, which happens to be in SAP or some thing sitting on top of SAP

[frank kayak]

thats just a harmless water pistol right?... hovers mouse over...

[simple ravine]

and they be like "yeah, that's on-premise behind 849032 firewalls and bullshit. so we need an express route connected to our api gateway"

yes, absolutely just water pistol

for the theatrical effect.

[frank kayak]

ah good

ill close the browser for now - firefox & poe doesnt fit that well into my 8gb - and when im not careful it just freezes the system - id prefer a crash of the program but no....

perhaps ill check discord later again

[simple ravine]

oh u dont use the discord app?

i was triggered by this btw

SAP Concur is a dumpster fire of its own, only nice feature is that the Amex card transactions gets put in there automatically

which kind of is a curse sometimes too

[worthy cape]

When @simple ravine plays PoE, he gets elemental ailment immunity just to not get afflicted with Sap.

[simple ravine]

hah hah

dear god, what have i done?

[worthy cape]

Don't cut yourself on the bleeding edge.

[simple ravine]

yeah, that was a bit too bleeding edge

[worthy cape]

^_^

[simple ravine]

the net6.0 target framework had existed for like 1 hour, and the binary bits didn't really work for it yet lol

i mean net5.0 is in RC1, not even final yet, and 6.0 to be released in 1 year.

[compact isle]

there was a spike but seems fine now

there was a console patch so probably just related to that going out

[frank kayak]

does anyone know if it would be allowed if the item prices in forum shops is automatically updated?

[worthy cape]

That was the whole purpose of tools like Acquisition in the past, wasn't it, to manage forum shops?

You're looking at more frequent automation twiddling?

[frank kayak]

acquisition but withg autopricing

forum shops instead of prem tabs because its outside the game

[simple ravine]

Well, forums is technically outside the actual game. Are there currently any rules that dictate how interactions can happen with the forums?

If you have to click a button that will "auto-suggest" prices and update the prices, vs a service that will do that for you in intervals is also a somewhat differentiating factor, maybe?

[frank kayak]

i would like the passive variant the most obviously but no clue...

[grave wren]

dont think theres anything written about forum interactions

[frank kayak]

at least i havent heard a rule related to thread posts or related to the website...

[pseudo ocean]

i think you might have to bump the thread

to upate

[pseudo ocean]

limit of once per hour for bumping your own thread in the trade section, so i think it limits your updates that way.

[rapid pagoda]

https://www.pathofexile.com/forum/view-thread/2965725 mentions the Freidrich contract. With this, I think all Heist content will have been revealed. Can someone confirm that there aren't any Heist quests which are currently disabled?

[velvet fog]

There are 6 unique contracts in total, and 2 unique contracts that have not been revealed yet.

Contract: The Slaver King drops Chains of Emancipation

Contract: Jamanra's Rest drops Corpsewalker

[rapid pagoda]

I'm pretty certain The Slaver King is the Freidrich contract mentioned in the announcement. Not sure about Jamanra's Rest.

[worthy cape]

I got nerdsniped into troubleshooting the CreateShaderResourceViewEx: Wrong Parameter error a lot of people with more potato machines experience.

After a lot of struggling, I find out that BC7 requires feature level 11_0, which aligns somewhat with the current minimum spec on Steam.

I should've known that already, but I blame that D3D11 has been around forever.

[pseudo ocean]

I think they wouldn't even be able to to boot the game if there dint have hw 11_0 though

Okay nevermind apparently dx9 hw gets dx11 api or something in windows 10

[frank kayak]

@earnest radish thansk for confirming my guess

[velvet fog]

@carmine merlin https://www.reddit.com/r/pathofexile/comments/j8crmt/pastebin_removing_path_of_building_exports_yes/

[worthy cape]

@coral ermine There's a distinction between the API version and the feature levels exposed by the hardware. From D3D11 (which is supported in Win7 (and in Vista with a platform update)), there's FL 11_0, 10_1, 10_0, 9_3, 9_2, 9_1.

The 9_* map roughly to D3D9.0a-c (with some stricter requirements), while 10_* are 10.0 and 10.1 verbatim, all with the D3D11 API. https://docs.microsoft.com/en-us/windows/win32/direct3d11/overviews-direct3d-11-devices-downlevel-intro

[pseudo ocean]

Ye but I thought that 11_0 was minimum required to have 11.0 api

And then 11_1 is optional

[worthy cape]

You always have the same API, it's just that some methods are not available. Even if you have a 9_1 FL you still bind resources with SRVs and all that.

[pseudo ocean]

Oh

Would your shaders even run though I guess maybe on some dx10 hw

[worthy cape]

When compiling the shaders you get to say what Shader Model it should target, you can build all variations and load the coolest one that is supported.

The capabilities are statically known offline to d3dcompiler, so you get hard errors on using too many registers or HLSL features not supported in the shader model.

So you might have a set of shaders built for like vs_4_0_level_9_1, vs_4_0, vs_5_1 and pick the one that the detected feature level supports, and assumedly also tune your pipeline accordingly.

[pseudo ocean]

I see, makes sense to target 4.0 too https://en.m.wikipedia.org/wiki/High-Level_Shading_Language

[worthy cape]

As for solutions to this, the only real one is to transcode the stored BC7 textures to a DXT format or even RGB(A) on load, which both costs cycles and VRAM.

Of course assuming that there's also not additional things that have changed that makes the game incompatible.

Happy I'm not holding that can of bees 🙂

[pseudo ocean]

I've heard that on mobile they target gles2.0 or maybe even 3.0 and there all the texture compressions are optional hardware feature

I guess they query before they download

Since most big games have in-app downloaders

[worthy cape]

I'm curious how far the client would get if one were to try transcoding and re-bundling all the resources, but I don't know how to get around the patcher validating things and have better things to do.

I just feel sorry for the people in the threads that have been left hanging since the tech patch.

[golden bane]

It looks like Pastebin started to delete anonymous PoB builds, so if anyone here wanted to make their own pastebin-but-only-for-PoB-builds website, now's the time 😉

[simple ravine]

that's unfortunate.

perhaps with some preview too would be a nice additional feature, hmm

wonder if it'd be possible to create c bindings to headless PoB

[worthy cape]

Finally on my way home from work, called myself in as everything was down this morning.

[simple ravine]

ouch

power outage?

[worthy cape]

Turns out that a Lot of things go wonky when the DNS server is upset.

Including auth and Kerberos

[golden bane]

I'd recommend embedding a Lua interpreter in your app and calling headless PoB from it instead

[simple ravine]

CoreDNS in K8s or regular?

I tried that, @golden bane, from .NET and it didn't really work out well. I know rasmus got it working, but didn't share the precise details on how.

Perhaps I should use another language for that particular set of feature if so.

But if one were to create something like that, it needs to be efficient, otherwise you risk paying too much for it

[worthy cape]

Regular BIND9 DNS with site specific gunk to generate zone configuration.

Super fun to break into your own infrastructure as things like KDCs are discovered via DNS

[golden bane]

@simple ravine Maybe try using a .NET <=> Lua bridge instead

https://github.com/NLua/NLua

[simple ravine]

I think rasmus had to satisfy a lot of wonky things to get it to work, in that headless example

If someone could help out with those things, I can help out with my cloud experience and get something fast and reliable up

There's likely a reason why pastebin are identifying PoBs and removing them... there are likely a lot of them made 😄

probably would want a TTL on those one way or another anyhow, as they kind of grow stale over a period of time. could extend TTL when they are accessed

[golden bane]

Probably, but that's an implementation detail

[worthy cape]

Host pathbin.com not found: 3(NXDOMAIN)

[golden bane]

MVP would be a basic URL shortener website, but for PoB codes. Preferably OSS, so you could rehost in case it's unmaintained. Analytics can always be added later.

[simple ravine]

sure, there's some interesting non-obvious caveats with URL shorteners, especially with high traffic

[golden bane]

I'd love to hear more about it

[simple ravine]

the tldr is that you got to pre-generate the keys with a smart algorithm and store them in a database, where you claim them

depending on the amount duplicate content, you likely also want to hash and reuse

[frank kayak]

does someone have experience with preparing pob for headless mode? i ran into a hurdle a while ago... cant really remember what it was

[grave wren]

hm i've read and used pb previously: https://github.com/ptpb/pb

seems kinda dead by now, thats a bit sad

[golden bane]

Useful for inspiration though, if were to do this myself at some point, I'd also use Flask

[grave wren]

the big question is should it be more than just a simple pastebin alternativ

or what's the longterm goal

[golden bane]

Longterm goal would probably be something like pob.ninja but for builds instead of characters

[grave wren]

basically poe technology (iirc)?

https://poe.technology/poebuddy/kwizDzkz

[golden bane]

Something like that

[grave wren]

seems useful, was thinking of building something like that from pastes i got from @regal mural bot

didnt have the time to do much yet tho

[golden bane]

I don't think I will have time as well, PoB is keeping me quite busy

[frank kayak]

@golden bane any tips on how tu run pob headless? i mean without the binaries, pure lua cross-platform

using github.com/yuin/gopher-lua

but there are other go implementations

[golden bane]

Never looked into it further than that you would have to bring your own zip utility

I'm happy to answer any questions about PoB's inner workings though

[frank kayak]

ok. when im continuing on that branch ill ask again 🙂

[grave wren]

would be interesting to see how many pastebin pob generates daily on avg

[simple ravine]

yeah, more than three fiddies, if pastebin is targeting them

perhaps they're doing a general cleanup though

[frank kayak]

the pure xml prob would be fine - that base64 crap though doesnt look good for pb

[golden bane]

Maybe it would help pasting the actual XMLs instead of garbled base64 endcoded stuff

[simple ravine]

i think so too

[grave wren]

i wonder what'd happen if you contacted them

[simple ravine]

they're gonna say "free service"

[golden bane]

Contacting them would be a bit awkward, as we're still using Openarl's API key

[grave wren]

oh he uses an api key, interesting

[frank kayak]

you should change the key...

[grave wren]

somehow thought it was using none with posts getting marked as spam

[golden bane]

https://github.com/PathOfBuildingCommunity/PathOfBuilding/blob/master/Classes/ImportTab.lua#L199

@frank kayak We should change it, but it wasn't a priority as everything was working until now. Even then, changing the key might not fix anything in this instance

[frank kayak]

its not an issue of if it was working with it - the problem its openarls key for his project - you dont just copy an api key - create a new one...

[golden bane]

We have explicit permission to use it

[frank kayak]

imagine someone doing dumb stuff on a fork - the api provider will go at the owner

[grave wren]

sheer volume of the shortening sounds kinda ew in general with the 24-50kb files, unknown amount of pastebins per month and then you somehow dont wanna pay out of pocket

[frank kayak]

ah ok

[simple ravine]

It looks like AlternateQualityTypes.dat has invalid references to Stats in StatsKey

anyone else seen or know anything about it?

[velvet fog]

no idea

[frigid nova]

In order to use PyPoE now do I need some auxilliary library to handle ooz?

[worthy cape]

@frigid nova either a stock ooz.exe or my adaptation into a DLL

Not sure how far Omega has gotten on their bundle handling, still WIP I believe

[pseudo ocean]

If the pastebins are 50kb you can fit like 20mil in an terabyte

I think that might be enough

@frank kayak I used the headless client a little not to long ago, you /need/ luajit I think you have to copy to some libraries ro where luajit can find them but besides that it worked when running the headlesswrapper.lua

[fickle tusk]

Longterm goal would probably be something like pob.ninja but for builds instead of characters
@golden bane would be interrested to help with that if someone eventually picks it up. Maybe iam going for it myself someday but currently i don't have enough time for that

[grave wren]

Problem is mainly keeping it cost neutral imo

[frank kayak]

the execution of pob in headless mode fails on line 215 for me

does the script need an argument?

i did extract the zips and replaced some c deps with pure lua that should offer the same interface

@pseudo ocean hi, so the luajit is only for discovering the binary dlls?

i have to do the execution without those on linux and macos

[pseudo ocean]

Na its syntac requirements I think

There are lua only dlls I think?

1 sec

So in the runtime-win32.zip there are some lua only dependancies in the lua filder, i think libcurl and lzip is just for the updater but it uses some of the other ones.

It uses a couple lua features that are exclusive to luajit, they wouldn't take long to fix, but otu of the box it just works with luajit

like you have to take the lua folder in the runtime-win32.zip folder put it in the root.

after that i think it runs with luajit

[frank kayak]

lcurl is prob not needed - i can feed pob the stuff from go, zip is also unneeded

[pseudo ocean]

Ye it doesn't use those in headless mode

[frank kayak]

my setup.sh imports https://github.com/kikito/sha1.lua

runtime-win32.zip is extracted by the setup

https://raw.githubusercontent.com/davidm/lua-bit-numberlua/master/lmod/bit/numberlua.lua is used as bitop.lua

iirc its compatible but slower to the c-using version

i then remove the BOMs from the lua files

find . -type f -not -regex '.*.(lua|cfg|txt|ini|xml)' -delete

its already a while since i touched this code

and this before starting headless: if err := L.DoString(bit = require 'bitop'); err != nil {...

[pseudo ocean]

I mean bits32 is just integrated in luajit I think

But you can wrap it so it it works with the bit library in later versions of regular lua, but they have slightly different api

I mean I think even, luajit with jit disabled is faster then regular lua

I tried to write a lua to go transpiler, but I got stuck at the type deduction/interference algos

Working on big lua codebases sucks

[frank kayak]

bit.* did make problems with pure lua

head of the numberlua.lua script i linked earlier:

[simple ravine]

from PyPoE.poe.file.specification.data import stable
from json import dump
with open('./spec.json', 'w') as f:
    dump(stable.specification.as_dict(), f)

Right now, I am doing the following, which requires a pip install, if I am not mistaken.

is there another way, where I can simply point to the stable.py physical file?

i.e. not having to pip install PyPoE

[vestal pond]

https://raw.githubusercontent.com/OmegaK2/PyPoE/dev/PyPoE/poe/file/specification/data/stable.py

[worthy cape]

If the deps are present, set the environment variable PYTHONPATH to indicate location of modules.

Or just stand in the right directory.

@vestal pond I believe he still wants to be able to load the module, which means using pypoe in-place.

[vestal pond]

Not need to install

[simple ravine]

yes, my goal is to create an automated ci/cd build

[worthy cape]

PYTHONPATH=$PWD/PyPoE python3 foo.py

[simple ravine]

ok cool, I'll try that

because I (programmer) am lazy and just want it to do it automagically 😄

[worthy cape]

We use PYTHONPATH extensively at work to swap out whole chunks of installed packages for side-by-side installations of almost everything.

[simple ravine]

ok

so what pip does is simply symlinking to a common place that is in my PYTHONPATH?

[worthy cape]

pip installs into one of the locations that are found in the site package lookup path.

PYTHONPATH augments the locations in which it looks.

See sys.path in a Python interpreter.

[simple ravine]

too in-depth for me 🙂

i just need a dumbed-down "do this and it works" when it comes to Python

[worthy cape]

zao@mim:~$ PYTHONPATH=/media/stuff/depots/tools/PyPoE python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/media/stuff/depots/tools/PyPoE', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/zao/.local/lib/python3.7/site-packages', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages']
>>>

Stuff you pip install --user ends up in the site-packages in your homedir, f.ex.

[simple ravine]

pip also installs a bunch of dependencies I guess

[worthy cape]

I'd probably just pip install in CI, tbh, makes your use of upstream less brittle.

[simple ravine]

hmm

[grave wren]

same

[simple ravine]

ok

the machine is ephemeral, so will have to do it each time

ugh, I will have to make some kind of polling mechanism

ok that's for later, distracted from main goal yet again 😄

[grave wren]

doesnt every ci/cd solution have either scheduling or polling

[simple ravine]

maybe, will have to check with either github actions or azure devops

to see if there's some polling that I can do on Omega's repo

not my specialty area, but interesting enough to figure out for sure

[grave wren]

dont you have auto polling if you just fetch the newest dev branch

[simple ravine]

hmm, u make me too curious, one sec

[grave wren]

pip also supports pulling from github so you can do that too: https://stackoverflow.com/a/24811490/1496040

[simple ravine]

looks like I can only select my own repos

and forks, but forks aren't updated automatically right?

[grave wren]

nope no auto update on forks, gotta add an upstream and fetch

[simple ravine]

trying to add it as a generic git repo, to see if there's a way to schedule it

[grave wren]

dont you just wanna have the dependency?

[simple ravine]

i want to spit out a new spec.json

every time it's updated

[grave wren]

ah

[simple ravine]

and then commit that to my repo, build etc

ah, well that kinda works.

better yet

[grave wren]

scheduler hasd a weekend break

[simple ravine]

don't machines need a break too?

😄

[grave wren]

but yeah polling is pretty basic wouldve surprised me if it was missing 😄

[simple ravine]

now I just need to figure out a good way to determine if stable.py was changed or something else, and then trigger only when necessary

[grave wren]

does your runner save the last commit hash it polled from?

[simple ravine]

i hope so.

i will have to try with another repo later, do some testing around

[grave wren]

what other ci solutions offer is a check if specific parts changed

so you'd only trigger the pipeline if that file changes

unsure if your solution has that

[simple ravine]

there's always this way

https://pumpingco.de/blog/run-an-azure-pipelines-job-only-if-source-code-has-changed/

[grave wren]

$response = (Invoke-RestMethod -Uri $url -Method GET -Headers $AzureDevOpsAuthenicationHeader)
$editedFiles = (git diff HEAD $response.sourceVersion --name-only)

is what azure offers apparently

hm interesting so no inbuilt support on azure so far

[simple ravine]

maybe github actions?

happy to mess around with that too

[grave wren]

unsure i only know that gitlab has it and jenkins

...
only:
  refs:
    - master
  changes:
    - src/**/yourfile.xyz

is how you'd do it in gitlab

https://github.community/t/is-it-possible-to-run-the-job-only-when-a-specific-file-changes/115484

Seems to be supported

[simple ravine]

i'm sold.

[grave wren]

glad i switched from travis to github actions as well

very comfy

[simple ravine]

I like how they added a cost limit to github hehe

in case things go haywire because i did a mistake lol

[grave wren]

😄

[simple ravine]

so u get 3000 minutes every month that should be enough I'm guessing

[grave wren]

isnt it completely free for FOSS? unsure tho

shouldve redone my tests for @regal mural earlier

very comfy that they run in 4s now

[simple ravine]

probably is yeah

github are almost too generous lol

[grave wren]

lazily started testing by not polling the pastebin content in advance

turns out RTT for 418 pastebins costs quite a bit

[simple ravine]

RTT?

[grave wren]

roundtriptime

[simple ravine]

what's the reason for not having them in LFS or just in the git repo?

[grave wren]

lazyness previously

https://github.com/poediscord/poediscordbot/tree/master/tests/in/pastebin_xmls now i have my collection

[simple ravine]

can be a good idea to re-evaluate, considering Cinnar's comment on pastebin culling the pastes

[grave wren]

yeah did that a bit ago already but just noticed how dramatically it cut time down

~5min => 4s

not the same machine but still roughly the same factor

was also doing a site that collected stats on pastebins the bot parses but with pastebin issues thats on ice for now 😄

[simple ravine]

i wonder how many pastebins that are created every day for pobs

[grave wren]

same, thats why i'd not want to start up any work on that without that knowledge

[simple ravine]

and some kind of collective funding of sorts

[worthy cape]

I wonder how much shit you'd get for PoB telemetry 🙂

[simple ravine]

because u never know how much that can spiral

hahaha, the outrage would be huge

could always proxy through something when creating pastebins

[grave wren]

they wouldnt dare sell my ballista love to advertisers

wonder if you have any insight on how much daily traffic your dev token uses from pastebin

[frank kayak]

why did it take 5min? were the tests sequential?

[grave wren]

it polled all 418 builds from pastebin and then evaluated

[frank kayak]

5mins seems long

[grave wren]

unsure if pytest does run sequentially tbh

hm apparently it doesnt

https://pastebin.com/doc_api#9

interesting, so you'd find out if you had access

[worthy cape]

@grave wren The limit of 1000 seems to screw you over tho.

[grave wren]

gotta schedule it but you'd find out how many you make per day

[worthy cape]

yeah

[frank kayak]

could someone with macos and installed chrome please test something for me? I need to know if the following command returns the chrome password (DO NOT POST THE OUTPUT HERE). Im not sure about the service and account strings and if the output has text around that needs to be stripped.

/usr/bin/security find-generic-password -s "Chrome Safe Storage" -wa "Chrome"

[worthy cape]

hunter2

[frank kayak]

?

[worthy cape]

(a joke)

The command pops an interactive dialog box for keychain password.

[frank kayak]

oh

[worthy cape]

"security wants to use your confidential information stored in "Chrome Safe Storage" in your keychain."

[frank kayak]

does it work after the popup?

[worthy cape]

Pops another box after the first

[frank kayak]

macos dev is just annoying

[worthy cape]

Then shows a base64 string on the console.

[frank kayak]

oh

then it works... with enough hurdles to make it not usable....

[worthy cape]

😄

[frank kayak]

and without the -s "..."

but reset the popup permissions that you got

?

[worthy cape]

I just put the machine in the bag again 😄

[frank kayak]

ah nvm then

thx for testing again!

[worthy cape]

Same prompts, same output.

[frank kayak]

thx :/

[simple ravine]

I am gonna be very nonchalant and just say 'Screw macOS players' with my potential tool(s). Sounds like a terrible boobytrapped ecosystem

[grave wren]

?

[worthy cape]

I wouldn’t say it’s a “boobytrap” to have a solid keychain that actually protects against snooping programs like yours.

[simple ravine]

😄

[frank drift]

how dare they not let you steal passwords

[simple ravine]

ooh, I thought this was for reading cookie sqlite storage

[frank kayak]

you mean as secure as those T2 chips?

@simple ravine yes it is

@frank drift the password i want is the generated password chrome uses to encrypt your cookies

its like having the doorkey-under-the-doormat-secure

[simple ravine]

hmm wonder if they changed it on windows as well

now i get curious

[frank kayak]

chrome 80 now uses DPAPI to encrypt a master key instead of the cookies directly

the encrypted masterkey is in the Local\ State json file

firefox 78 also changed the db layout

[simple ravine]

ah yes

[frank kayak]

@simple ravine use this as ref for the new chrome 80 windows enc: https://stackoverflow.com/a/60423699

if you could decrypt older windows chrome cookies (DPAPI) you can use that method for the masterkey

[simple ravine]

win32crypt.CryptUnprotectData

this is what i am doing

[frank kayak]

me too

[simple ravine]

ah..

theyre padding the key

for whatever reason

[frank kayak]

@worthy cape im not that sure if the login keychain pw queries normally popup - i did see it used in libraries and it would be weird if a lib func call would trigger a popup

padding?

[simple ravine]

in windows, there's a user contextual thing as well

you can't unprotect this using another user as an example

encrypted_key[5:] does theat mean skip 5 in python?

or take all but 5?

[frank kayak]

yeah its local for the user

those 5 are "DPAPI" iirc

start at 5th pos

like in go

[frank kayak]

now i can hack your system!!1!!

[simple ravine]

hehe ok.

gl

[frank kayak]

im in

[simple ravine]

so slow.

well, mine doesn't have any DPAPI in it interestingly enough

[grave wren]

time to upload all the discworld novels and delete the tech stuff

[simple ravine]

ah im sloppy nvm

[frank kayak]

it does

base64 -d first

[simple ravine]

yes.

[frank kayak]

i saw that RFBB... prefix too

[simple ravine]

e.g. <-- sloppy

[frank kayak]

ah and i cant read

[simple ravine]

getting there

[frank kayak]

yup

[simple ravine]

before, I just did Unprotect on the encrypted_value in the sqlite table

was a lot more convenient

[frank kayak]

the call is a bit slow i think, chrome handles the aes enc now itself

[simple ravine]

works!

I have an issue though, that when Chrome is open, it's waiting for Chrome to exit before reading

that's kind of annoying

[frank kayak]

file access stuff

works on linux though

[simple ravine]

I'll see if there's a "read only" mode one can use

[frank kayak]

flash plugin did delete the videos in the past while still having access onto them...

[simple ravine]

kek, just file copy the sqlite read and then delete bak

[frank kayak]

havent even checked what my code does on windows when the browser still accesses the db

[frank kayak]

Is "awakened poe trade" compliant with the tos? there is this map check and delve grid im not sure about what its supposed to be.

[frank kayak]

map check seems to be nice bosss arena screenshots

delve grid is a raster overlay over the delve map i guess? are such overlay allowed?

[inland kestrel]

@polar island i noticed that sometimes the right side shows the total energy shield and other times it doesn't. is this because it's jewelery? or because of certain mods? It's helpful to see the total es on an item after all the flat/inc/intelligence/quality/etc so I wasn't sure if this is an easy display chance you might consider

• shown: https://i.imgur.com/OBB5pKi.png
• not shown: https://i.imgur.com/UFXKRza.png

[inland kestrel]

I had a feeling that'd be the case. I have a broad live search that isn't specific to ES items. I want to see the total ES without constraining myself to items that have a non-zero ES. Is that possible with that filter?

[inland kestrel]

looking at that link,
https://i.imgur.com/0rlxXnh.png

is the ES 83? or 69?

awakened poe says 82, lol
https://i.imgur.com/jRgbfgK.png

take this as an example https://poeapp.com/search/Heist/MN2LSY
i'd like to see ALL items that match the search (a contrived one here), not just items which have ES as a property

here is a better example: crystal belts usually have ES. but they don't have an ES property
https://poeapp.com/search/Heist/W8pfZX

[inland kestrel]

well, it's not estimated quality, right? it's specified quality

I see what you meant by quality now; on the mind cage example above, awakened poe trade shows the ES if quality was 20%. A calculation you could potential show but that wasn't my initial ask

yup

maybe my immediate concern is... why don't belts (a crystal belt for example) have an energy shield property?

https://i.imgur.com/v7L2pDX.png
should be what... 155 ES ?

[violet path]

isn't this just how it's presented in the trade api stuff? armours/weapons have base stats like ar/ev/es that is listed differently from implicits like on crystal belts?

[inland kestrel]

yeah, i'm disappointed with how it's represented in-game. that's why I use your app 🙂

that data seems like it's all there (for the site to derive the property)

but i can't sum +max es and increased % es to get an accurate number comparison (between items)

[inland kestrel]

❤️ i know it's a feature request. appreciate you hearing me out

[rare lotus]

hey, just curous - how are you storing all the poe item data, elasticsearch?

[velvet fog]

GGG use PostgreSQL

[worthy cape]

@frank kayak I don’t know if I have any additional password set for my login keychain or it’s a legacy behavior, but I’d kind of expect it to happen if you try it from outside the program’s context.

[pseudo ocean]

I think poe.xyz is elastic search

But he probably deals with a lot more string data

[rare lotus]

thats gotta be a beefy postgres database

I think its gotta be elastic, not sure how to manage all those items + searching otherwise

[simple ravine]

shards and read replicas

[pseudo ocean]

you can do anything with postgres if you know what you're doing

i've heard

just need to be a DBA Demigod

[frank kayak]

anyone that has meddled with dbus?

have problems attaching signals with godbus

[simple ravine]

maybe it's a divine intervention ^_^

[fickle yew]

@simple ravine I have a short alpine Dockerfile for extracting the spec.json if you need

[simple ravine]

@fickle yew that would be nice yeah

[fickle yew]

@simple ravine https://gist.github.com/rasmuskl/e7bcd2ab2a07a60b0734ded0c197e87e

[simple ravine]

thanks!

[frank kayak]

are tools allowed that display an overlay on top?

i cant remember any concrete statement for this

[grave wren]

On top of what

[frank kayak]

over the game without any manipulation

just an extra layer

[worthy cape]

The traditional tool overlay is a transparent window with clickthrough, but that's mostly on-demand.

You can't hook to inject one.

[frank kayak]

awakened poe trade has a grid for delve i think - havent tested it

or stuff like stash overlays that highlight the position of an item

[worthy cape]

If you're running exclusive fullscreen you're SOL.

[frank kayak]

SOL?

[worthy cape]

Shit Out of Luck.

[frank kayak]

why?

[worthy cape]

Can't have windows over an exclusive fullscreen program, and you may not hook.

[frank kayak]

i do run poe in full hd fullscreen mode with wine explorer

[worthy cape]

If you're in hippieland, none of the above applies ofc 🙂

[frank kayak]

yay

[worthy cape]

Well, you're still not supposed to fuck with the client.

[frank kayak]

but awakened is ok right?

[grave wren]

Likely it's around for a while now and ggg hasn't said anything negative about it

[frank kayak]

i only parse client.txt atm without making use of it and have ideas of using CV or only OCR later on perhaps in my own tool

@grave wren ok - im just cautious with 3rd party tools

most didnt even work on linux - only ever used pob & emmitts planner...

ill try it out then

[frank kayak]

i need another poe tool where i can scrape the poesessid besides acquisition and procurement. any suggestions?

[simple ravine]

im curious, why?

[dull laurel]

scrape? you can get them from the browser, once you are logged in

[simple ravine]

ah getting there.

now accessing from raw bytes, rather than doing a bunch of hoola-hoops

those 26ms is for reading all the mods rows btw, not just the top 10

[velvet fog]

@carmine merlin ActiveSkillType.Type96 seems Second Wind Support https://poedb.tw/us/Second_Wind_Support

[swift beacon]

Maybe this is a silly question, but does anyone by chance have a copy of the GGPK from 3.11.2? I'm kicking myself for not having saved a copy.

[rapid pagoda]

How specific of a version are you looking for? I've got a couple of GGPKs from 3.11 but not sure if I got 3.11.2 precisely

[swift beacon]

I need one that is after the new bundle system was added in 3.11.2 but before 3.12 / Heist.

[carmine merlin]

@velvet fog thanks for letting me know. I’ll change it for the next update

[worthy cape]

@swift beacon I might have one copied lying around, but the full set I keep are sourced from Steam which are loose files now.

[swift beacon]

Okay, that's great if you have one otherwise no biggie.

[worthy cape]

Seems like I deleted my standalones just a few days ago for disk space unless there's one lurking in some working directory somewhere.

[fickle yew]

@simple ravine nice performance numbers 🙂

[worthy cape]

@swift beacon What kind of thing are you looking for in it? The Steam flavour for that version is easily obtainable, as are every single release since 1.0.0.

[swift beacon]

Really, how do you get those? Does Steam archive them or something?

[worthy cape]

@swift beacon Yeah, you can obtain any specific manifest you want from the Steam API with the right tooling.

https://steamdb.info/depot/238961/manifests/
https://github.com/SteamRE/DepotDownloader/

The tool is slow for anything pre-3.11.2 as it doesn't handle large files well, but fast enough for me to obtain every single one over summer.

[signal briar]

sry i didnt know how where to ask... is that possible to check with a programm if someone trade? or create a party

[worthy cape]

@signal briar Nope. Only things that are logged into Client.txt like whispers.

It's rather against ToS to read the screen and react to that.

[signal briar]

okay because we want to create a HCSFF heist league but on the website is dissapear that option

[swift beacon]

@worthy cape Whoa that's awesome, thanks.

[worthy cape]

@swift beacon The only annoying thing is mapping a manifest ID to the actual PoE release. You can look for strings inside the EXE to get the little version number shown in the lower right of the login screen, but that lacks hotfix information.

[frank drift]

@signal briar did you contact GGG support about why you can't create hc ssf league?

[signal briar]

yeah they said only right know is impossible (the reason )/shrug

they didnt said anyreason

[worthy cape]

@swift beacon ```
tags/3.11.2 336757500521109937
tags/3.11.2b 6105810828087164174
tags/3.12.0 1106766371751316230
tags/3.12.0 1758646762025641699
tags/3.12.0 1871807131643013746
tags/3.12.0 2420864645308471346
tags/3.12.0 7035527581453360894
tags/3.12.0 7358046278015163815
tags/3.12.1 3341589452991106600
tags/3.12.2 1839254412846430668
tags/3.12.2 4642425847506640743
tags/3.12.2 5854101082538693936
tags/3.12.2 6248188790027187970
tags/3.12.2b 7037703177661217284
tags/3.12.3 3653104038963790613
tags/3.12.3b 4570085118665568963
tags/3.12.3c 4114030591192467865

The command is something like depotdownloader -app 238960 -depot 238961 -manifest $manifest -username $username -os windows -osarch 64 -remember-password -dir downloading/$manifest

[simple ravine]

@fickle yew I think I can get rid of all the boxing, by having the API surface indicate expected type

that would make it preeetty quick

[worthy cape]

It can only run one instance at a time unless you mess with the integer login ID.

[simple ravine]

and unboxing of course

[worthy cape]

@swift beacon I've obtained all of them in case they're needed for file format exploration or lore stuff, but my tooling is still a bit hosed 😄

The total undeduplicated dataset is something like 10TB+

[simple ravine]

inb4 someone says: bUt sTORagE iS cHEaP

[swift beacon]

That's a lot lol. I have about a dozen past versions saved, but that's probably more than I need already. Except for the occasional times I need a specific one and don't have it...

[worthy cape]

I've delta-compressed them so I keep the last GGPK in each league and have reverse patches to get older ones.

I also have a format where I store the files in a sqlite3 database by content hash and have modified GGPK files that just contain the tree and no payloads.

That's just 144 GiB for the DB and 19 GiB for the indices.

Haven't adapted that for the bundles yet, so just all the contents up until the tech patch.

The delta-compressed dataset is just shy of 800 GiB.

[swift beacon]

Well I guess if GGG somehow loses all of their backups it won't be a problem 😄

[worthy cape]

I've been super behind on tooldev since league start, but as my build is starting to run into a wall, maybe maybe I'll get something done soon again.

@swift beacon Do you have tools to work with bundles or do you need something horribly hacky like my bun-extract-file tool?

[swift beacon]

I wrote my own and it's working well so far. I can handle most of the file types including the .dat files.

[worthy cape]

Ah, great.

[swift beacon]

Haven't quite figured out the .dat64 files yet though, but doesn't seem like it will be too hard.

[worthy cape]

@swift beacon Same contents as .dat but wider datatypes.

.datl is same as .dat but with UTF-32 text.

[swift beacon]

Not sure if it's even really necessary to handle them, since afaik they have the same data as the .dat version?

[worthy cape]

See table in https://github.com/OmegaK2/PyPoE/issues/134

[swift beacon]

@worthy cape Yeah, I'm running into something where it seems like each row has an extra 8 bytes at the end that's always 0 for the tables I've tested.

[worthy cape]

Anything that's pointing into variable data, referencing a different row, or referencing a row in a different DAT is wider.

Speculation here is that on load they are resolved into being raw pointers.

[swift beacon]

I'm sure I'll figure it out, but it hasn't been much of a priority since I can just use the .dat files.

I'm more concerned with figuring out the new texture system for the newer models, since they don't work well with my model viewer.

[worthy cape]

Textures are the same but are BC7 instead of DXT formats.

This means, among other things, mandatory D3D11.0

At least as far as I've seen, if you've spotted something off I'd love to hear about it.

[swift beacon]

I can handle the images, it's more interpreting the .mat files.

[worthy cape]

Ooh, they changed those again?

I have yet to interpret the FX graphs properly.

[swift beacon]

They have a version number and most of them are verison 3. But the newer ones are version 4 and the format is totally different.

[worthy cape]

Isn't v4 the JSON format that has been around for a long while?

Ah yes, it's the JSON - https://gitlab.com/zao/poe-rs/-/blob/master/src/formats/mat/types.rs

[swift beacon]

Yes, it's the JSON one. I think it started around Betrayal, so I guess they're not really "new" any more.

[worthy cape]

I got it parsed and moved onto more urgent formats, not being keen on reimplementing the whole pipeline and hooking in the material noodle soup.

[swift beacon]

I have a model viewer that works really well with the version 3 .mat files, but the version 4 ones seem to use more complicated rendering and I don't know that much about 3D models.

[worthy cape]

https://i.imgur.com/z4WNtyj.png https://i.imgur.com/KwQGlqK.png

Lovely soup 😄

[swift beacon]

I'm just trying to get them look okay without having to worrying about lighting and shadows and layering and all that.

Yeah exactly.

[worthy cape]

My plan has been to try to identify the building blocks that seem to be built-ins, make a half-arsed pipeline from that, and then generate naive shader source matching all the noodles.

But again, sidetracked, and now changing languages from Rust because of reasons.

Seems like you're on a similar journey as me, my ultimate goal is hideout viewing and editing, but there's been many alluring formats along the way

[swift beacon]

That would be cool. I've looked at trying to get the environmental models working with my viewer, and I started trying to figure out the .tgm/.fmt etc files, but I didn't get very far.

[worthy cape]

I don't think fmt is too far from smd in shape, I haven't documented it in detail but have a writeup on smd at least and some notes on how meshes fit together courtesy of Chuan - https://zao.github.io/poe-doc/meshes.html

My current fmt parser is at https://gitlab.com/zao/poe-rs/-/tree/master/src/formats/fmt

parse_fmt should be readable enough if you squint 😄

010 Template for fmt: https://gist.github.com/zao/4589e63c464105ace3ae6bd9ab47324b

tgm I barely got started on.

[swift beacon]

Nice, I'll take a look at it tomorrow. Time for me to get some sleep. 😴

Thanks for all the pointers.

[worthy cape]

[frank kayak]

@simple ravine my tool searches automatically for a valid POESESSID after giving it the ok in firefox, chrome, acquisition & procurement - i thought it might make the entry easier for newbies who cant get the cookie out of the browser - now its mostly fun 🙂 - i do like scraping a lot

[velvet fog]

POESESSID is unsafe for public use, GGG should provide OAuth if they have time

[frank kayak]

arent there services that do use oauth with pathofexile.com?

i think some things were restricted though with oauth making it not usable for that harvest webpage last league

[worthy cape]

Exilence Next is one of the freestanding clients that use oauth, but that's mostly stash-wise.

[frank kayak]

ah

[frank drift]

filterblade uses oauth

[frank kayak]

novynn's tool does too iirc

was annoying to compile on Debian stable with those current Qt deps

https://github.com/Novynn/Adamant

i do like Qt GUIs just not compiling them...

[frank kayak]

@worthy cape could you when you have some time again please check something on macos? the GUI I use (nucular) for linux, windows, macos, android has 2 backends: shiny & gio. gio is for macos the default but requires cgo which makes cross-compilation a lot more difficult. I just noticed that the shiny backend also compiles for macos but i cant test if it actually runs...
compilation command:

CGO_ENABLED=0 GOOS=darwin go build -tags nucular_shiny

CGO_ENABLED=0 makes sure that the go compiler doesnt use a c compiler and the tag switches the gui backend.

[latent solar]

has there been any final stance from GGG about timer trackers such as mercury trade's overseer?

[worthy cape]

Not sure if I have any time before the end of the workday, so many meetings.

[frank kayak]

@worthy cape no need if you dont have the time & thanks a lot! - im simply interested if shiny is the alternative i could use on macos or not

[frank drift]

@latent solar unlikely to be an official stance unless pressed, but they have bigger fish to fry than overlays

[worthy cape]

@frank kayak builds, poem-tmp gui does nothing, zero exit code.

[frank kayak]

ah damn, thanks a lot for the testing

[worthy cape]

@velvet fog Hrm, is poedb.tw down?

[primal schooner]

what the minimum number of arguments I need to pass to the public-stash-tabs API endpoint in order to return the desired results? I'm just trying to filter by account name...

or is there any documentation on the API usage? (that's my real question, lol)

[worthy cape]

@primal schooner Not sure what you mean by "minimum number of arguments".

You give it a change id (which are offsets into each of the shards or something), get a bunch of stashes, and a next ID. You have no choice but to iterate on that.

Notable is also that the stash tab river doesn't contain historical state, older states for a particular tab are obsoleted and only the most recent one will appear as you iterate.

There's no filters, only the change-id parameter.

This isn't an endpoint to query the current state of some particular set of stash tabs (like ones belonging to a player). It's a stream of every single public stash tab in the realm.

[primal schooner]

oh, so, I'd have to parse the entire JSON and then filer/pick out the parts I'd want programmatically after that

I can't jsut return stashes belonging to accountName

[worthy cape]

To follow the river you find a change ID from which you care about things (like the start of a league or right now), grab that and pick out the information that might be relevant, use the next_change_id from that and go again, honoring rate limits.

poe.ninja has the "current" change ID in the stats page - right now approximately https://www.pathofexile.com/api/public-stash-tabs?id=844385955-856929072-819314315-924669656-884011905

@primal schooner So what are you trying to do anyway, might be a better question?

[simple ravine]

dafuk.

[worthy cape]

There are other authenticated endpoints to query your own stash contents, for example.

[primal schooner]

yeah i see that. so i'm using the Chaos Recipe Overlay git as a starting point

and I bascially want to create a dump-tab pricer

so everything you put into a designated tab is evaluated against like items in trade and given an estimated price (according to the methodology/algorithm I've developed)

Chaos Recipe Overlay uses Session ID and get-stash-items endpoint to generate the local list

[worthy cape]

Maybe something like poeprices.info's estimator (like what APT and other overlays use) is a better choice?

Or do programmatic trade searches, again like those tools.

[primal schooner]

that's the endpoint and methodology i'm assuming i'm going to be using to get teh local list

[civic crane]

@primal schooner so you want Exilence next with ability to price check rares?

[simple ravine]

Awakened PoE?

[worthy cape]

Note that the endpoints are all rate limited to some extent and if you fetch all stash tabs, you may throttle quite hard.

[primal schooner]

IDK what Exilence Next is

this is my dream

You use an overlay to designate a dump tab

and you have the overlay pull out over to the side or wherever

[simple ravine]

what do you mean designate a dump tab?

[primal schooner]

The when you open the overlay, it has the list of items and their values

you use something to designate the tab

like a local client, or web client

[simple ravine]

That shouldn't be too hard to create

[primal schooner]

a setting in the overlay

yeah it's not

the problem is the parsing the item attributes and then, parsing like items from the api, and then correctly evaluating them

and then the little down arrows are dropdowns that let you see what attributes are giving you that assumed value

based on a better 'top-down' methodology of pricing items (rares specifically)

[simple ravine]

thats what u have unit tests for

tdd

add some live unit testing, and you will have a tight feedback/dev loop which should help you out

[primal schooner]

rarity>type>base>ilvl>#socs>#ofLinkedsocs>colorsocs>Influences>enchants>implicits>affixes>statuses> etc

[simple ravine]

poe-prices.info does that... i don't think they're very accurate, but i doubt you'll do better unless you spend tremendous effort.

it's at least an indication to what's interesting to check closer, i.e. create programmatically some trade search to check further

[primal schooner]

Time and Effort I have in ample supply. knowledge of this type of dev is not.

[simple ravine]

well, either you leverage their thing, and hope for the best, or go get learning.

fair warning, it's a deep rabbit hole

you'll call yourself alice for a significant amount of time

[primal schooner]

yeah, I figured i should probably learn node anyways

considering my devs do so many API projects in it. lol

[simple ravine]

enjoy

[primal schooner]

I own a web application development company and haven't written a production line of code in 4 years... 🤷‍♂️

Electron is best way to do POE overlays ?

[simple ravine]

subjective question.

[primal schooner]

most used?

[simple ravine]

i don't have the data. most used != best

easiest? maybe.

suitable? perhaps.

[primal schooner]

yeah, i want the "wordpress" of POE overlays

support is my biggest concern

[simple ravine]

what's ur website?

[primal schooner]

don't want to start using something nobody else uses

[frank drift]

electron and similar embedded web frameworks are the most popular way to make desktop apps in general right now

[simple ravine]

based on what data?

[frank drift]

PoE overlays tend to be whatever the author already knows

well what is Discord

[simple ravine]

is that data?

[frank drift]

yep

[simple ravine]

Ok, you're referencing 1 data point. Cool story.

[frank drift]

thanks

name a popular desktop app from the last few years that's not electron or similar embedded web framework

[simple ravine]

That'd still be a novelty, not based on facts or figures.

[frank drift]

if you look at other games like League of Legends, the League client itself is an embedded web framework and pretty much every overlay for it is an embedded web framework, mostly electron

[simple ravine]

Yes, but not Electron.

CEF.

[frank drift]

practically the same thing

[simple ravine]

Spotify, Netflix App (desktop), Audible, Most Adobe apps and other creative applications, including Blender, Cinema 4D, Houdini, Autodesk apps, Office, Wunderlist, Todo, Twitter for Desktop, Evernote desktop

[frank drift]

spotify is CEF

again dissembling

netflix is almost certainly some sort of embedded web app also

[simple ravine]

Oh, didn't realize Spotify was CEF. Interesting

Eitherway, it's a moot point. Without data backing the claim up, it's not worth much in my opinion.

[frank drift]

you're the one making the claim that embedded web frameworks aren't what most apps are using

your opinion is worthless, you're providing no data

[simple ravine]

I have not claimed that.

[frank drift]

anyway if thezensei is done dick-waving, @primal schooner certainly you won't have problems with electron being unpopular, but you're getting ahead of yourself probably

worry about your own motivation to develop your project first

[violet path]

🎆

[simple ravine]

How am I d**k waving? I asked for further information. I prior to that have not claimed the opposite of what you claimed, nor advising against Electron specifically.

Yet you're saying I did say something I didn't...

[primal schooner]

@frank drift i'm very motivated to create it. assessing how much effort / ability it will require and subsequently deciding if that is worth it is a different story. lol

[primal schooner]

Is there documentation for any of the endpoints?

if so, where would I find it?

[minor charm]

@primal schooner https://app.swaggerhub.com/apis/Chuanhsing/poe/1.0.0

https://www.pathofexile.com/developer/docs/api-resources

also check the pins on this channel

[primal schooner]

thanks!

[frank kayak]

i hate that electron memory hog...

who had the idea that embedding a browser is a good idea?

[frank drift]

web companies

don't have to make a real desktop app when you can just dump your webapp in an embedded browser and call it done

[simple ravine]

buT iT'S sO pOPuLAR

jk

[frank drift]

you're right it is popular

definitely has its issues

feels like embedded chromium has global bottlenecks, the more electron/CEF/etc apps I have running, the more frequently they all lock up at the same time

[dull laurel]

maybe they could save memory by reusing the same chromium runtime? instead of all of them shipping their own full blown executable

[worthy cape]

That's quite the distribution headache tho.

It "works" on phones, with surprise breakage when the shared runtime has bugs or makes breaking changes.

I'm fine with Electron and friends if it means that I can use collaboration tools on hippie OSes.

[tiny cargo]

maybe they could save memory by reusing the same chromium runtime? instead of all of them shipping their own full blown executable
You mean like a program with a runtime that can "browse" different applications, maybe even has some addons and so on? Going fullcircle

[frank drift]

google pushed an "app" feature for Chrome for awhile, but seemingly it was pretty unpopular and got depricated

[simple ravine]

Not sure how much headache Electron/CEF has as a runtime. I think why these apps have gotten so notorious, is that web developers aren't really known for trying to keep the memory allocation down etc.

I mean... just look at all the package dependencies in that ecosystem for various things, and the architecture around some of these frameworks...

Not surprised when sorting on memory consumption, the 5 of 6 are web/electron/cef stuff

[rugged night]

Stashtab API:
I have noticed the following types of values in properties/requirements:

"values":[["28-53",0]]
"values":[["1.50",0]]
"values":[["7.00%",0]]```
So... if I'm understand this right its an empty array... or a single array entry (which is a nameless array of 2 values).

Does it ever deviate from this format?

For instance... are there other amounts of arrays in some items (or always 1 entry with an array containing 2 values)?

[velvet fog]

Flask have 2 args

[rugged night]

Ahhh... I see. Anything with "displayMode":3 has 2 values.

Thanks 🙂

[velvet fog]

not really

"displayMode":3 means fmtlib string in it's name

displayMode
0 means name should go before the values.
1 means the values should go after the name.
2 is a progress bar (for skill gem experience).
3 means that the name should have occurances of {0}, {1}, etc replaced with the values.

[rugged night]

Ok... so anything with "displayMode":3 CAN have multiple values, but might not? (other display modes can only have 1?)

[velvet fog]

Lasts {0} Second
Recovers {0} Mana over {1} Second
Requires {0} more Adjacent {1} Seeds of at least Tier {2} to Grow

[rugged night]

Thanks 🙂

[velvet fog]

"values":[["28-53",0]] and 0 is color

[civic crane]

🤦‍♂️

[worthy cape]

😄

[velvet fog]

@civic crane BaseItemTypes.dat::ModDomainsKey point to different stat_descriptions

[lean spindle]

Does anyone know which data file maps Atlas map nodes to their item image?

[rapid pagoda]

AtlasNode references ItemVisualIdentity.

[lean spindle]

Ah! it does. I somehow overlooked that. ty!

[obtuse citrus]

the file is just the icon btw

you'll need the base file from the mapseries

and also change the colour

[lean spindle]

Yeah, I've already put something together to do that part. Right now I'm just trying to automate the process of generating new files for each league.

(to this point, I've manually copied the images I needed for my project)

[rapid pagoda]

What's the project?

[lean spindle]

https://poeatlas.net

[rapid pagoda]

Oh neat! Feel free to steal ideas from my (much uglier) version, https://oriath.net/atlas/

[lean spindle]

Ooh, that's a neat progression tracker! I especially appreciate the filtering options you have at the top right. Very useful

[simple ravine]

Is there supposed to be an atlas image in oriath.net/atlas?

[worthy cape]

Yay, the bare hideouts were finally released. Fewer things to spoil now 😄

[simple ravine]

my wallet gonna hurt now?

[worthy cape]

Celestial Nebula but different tiling ground, 145 points each.

[grave wren]

Only if you love badly tiled hideouts

[simple ravine]

ah, I use the Celestial Nevula one now

kinda like it

[worthy cape]

Nothing tiled plainly is going to look good, apart from maybe the water/cloud one.

[simple ravine]

because I spent 200 shaper runs not getting the actual hideout I wanted

[worthy cape]

That's why you cover it in tons of decorations 🙂

[simple ravine]

ya, but I feel I can't buy all the cool decorations, even though I have spent 4 years in PoE getting max levels on all the people

but my lvl 7 Haku doesn't really matter anymore

apparently I need lvl 7 <insert-new-npc>

I have a decent amount of favor

[grave wren]

do any of you have a list of all available monster names in poe by any chance

[swift beacon]

I think it's MonsterVarities.dat. I can export it to Excel, but I'm not sure how I'd send it to you.

Can you post files in discord?

[worthy cape]

Yup.

[grave wren]

would be nice, i currently dont have access to my ggpk

[worthy cape]

All my infra is in shambles 🙂

[grave wren]

awesome, thanks

[swift beacon]

👍

[grave wren]

which of the tools is currently up to date btw, mostly used pypoe gui - does that still work with the new ggpk?

UI will be reworked for bundle support and is not functional at the moment

aw

[worthy cape]

I use my Bun stuff to extract files, but no DAT shenanigans.

You could technically repack unbundled DATs and use PyPoE still I guess.

[grave wren]

not attached to any tool i'd just like anything that works with the new format and provides some form of easy access to export data

[frigid sierra]

did anything change in the API since yesterday or so? my trade queries are all returning error 1020 now

[worthy cape]

What endpoint is that and if it's trying to use the trade site itself, how complex are the queries?

[frigid sierra]

the https://www.pathofexile.com/api/trade/search/ endpoint, the queries don't seem to matter, even the simplest one returned the error

it has been running fine for months, but suddenly broke yesterday

[worthy cape]

So queries like https://www.pathofexile.com/trade/search/Heist?q={%22query%22:{%22filters%22:{},%22type%22:%22Diviner%27s%20Incubator%22}} ?

[frigid sierra]

I use POST instead of GET, but yes

[frank kayak]

i would copy the behaviour of the webpage

[frigid sierra]

the webpage uses POST

[rapid pagoda]

@simple ravine Nope, no images on /atlas/, just text 🙂

[sonic horizon]

@frigid sierra I am using Poetradesearch EN translated by Redsol and having the same issue

[frigid sierra]

I found the issue, I need to add a bunch more headers into the request to make it look more legitimate. Looks like cloudflare was dropping suspicious requests with some missing headers.

[sonic horizon]

I might need to switch to a new thing, I don't have the tech skill to repair it, and he hasn't worked on his file in forever.

[young osprey]

Is there any sort of issue tracker for poe.ninja? I have been looking a bit and have not found one.

[frank drift]

I don't think a public one

if you report issues/suggestions to rasmuskl he's receptive

[young osprey]

Ok, I will when I see him online again

[simple ravine]

rasmkuskl*

[worthy cape]

I finally got my arse in gear today and got working on my VFS again.

[simple ravine]

I am trying to negotiate with the JITer

[worthy cape]

I had forgotten how miserable it was to index the old GGPKs.

"why isn't my code instant?"

[simple ravine]

a "few" more files

[primal schooner]

what endpoint(s) does Ras use for builds? seems like the only one that gets characters requires an account name...

[grave wren]

ladder api most likely

[primal schooner]

kk i'll take a look at the docs on that endpoint

thnx

[primal schooner]

these docs at https://app.swaggerhub.com/apis/Chuanhsing/poe/1.0.0# are not fully fleshed out, are they? seems like there's more params that some of these endpoints can accept

if you pass the ladders endpoint a class param (int) it returns all the ascendancies for that class (1=marauder, etc), and yet it calls the ascendancies 'class' in the output JSON...

is there a rule of thumb for ascendancy classification that these endpoints have that anyone is aware of?

[simple ravine]

well, Berserker is type of Marauder..

naming is hard 🙂

[primal schooner]

right class 1 returns berserkers, chieftains, and juggs

which are the ascendancies of the marauder "class".

i was hoping I could return by ascendancy

considering that's what are in the field "class' in the return json

[simple ravine]

Anyone who can with pseudo-logic give me some insight to reading ref|list|ref|string

my logic seems flawed.

[worthy cape]

Read the two words of offset and count, from var-data at offset read count words, each referencing a string at var-data offsets.

[simple ravine]

right yeah

same as ref|string but N times

[primal schooner]

other than the account / character params, what else do you need to pass the api/character-window endpoint to return data?

[worthy cape]

Note that it's a list of N offsets in the var-data, not N string literals back-to-back.

[simple ravine]

oh

that's why

[primal schooner]

I'm assuming it's something sessionID related that I keep getting error code 1'd

[civic crane]

😅

[worthy cape]

Expertly drawn 😄

[simple ravine]

pro diagrammer 😄

aw yeah, got it to work

public string[] GetStringList(string col)
{
    var fld = GetFieldForColumn(col);
    var bytes = _parent.Bytes.AsSpan();
    var buf = bytes.Slice(fld.Offset + (_rowIndex * _parent.RowSize), fld.Size);

    var elements = buf.ConsumeTo<int>();
    var offset = buf.To<int>();

    if (elements.IsNullValue() || offset.IsNullValue())
        return Array.Empty<string>();
                
    var data = bytes.Slice(_parent.DataOffset);

    var dataOffsets = data.Slice(offset, 4 * elements);
    var offsets = dataOffsets.Cast<byte, int>();

    var strings = new string[elements];

    for (int i = 0; i < elements; i++)
    {
        var df = data.Slice(offsets[i]);
        var strLen = df.IndexOf(DirectDatValue.StringNullTerminator);
        if (strLen % 2 != 0)
            strLen++;

        strings[i] = df.Slice(0, strLen).ToUnicodeText();
    }

    return strings;
}

kek

but it's fast!

[worthy cape]

That looks sketchy, isn't there a false positive in code units that have 0x00 in the lower byte?

[simple ravine]

            strLen++;```

[worthy cape]

What's StringNullTerminator?

[simple ravine]

u mean this?

0x00_00_00_00

[worthy cape]

There's legitimate occurences of 0x00 0x00 in the input.

I don't know what the reasoning is about the weird logic in PyPoE around strings.

[simple ravine]

hmm

Do you know an example I can make a unit test out of?

[worthy cape]

Are you/they trying to say that strings are always double-terminated with two 16-bit NULs?

[civic crane]

yes) otherwise how you will find end

[simple ravine]

As far as I can tell, strings are terminated by 0x00_00_00_00 i.e. 4 NULL bytes

[worthy cape]

My parsers always just scanned linearly in 16-bit quantities and stopped on 0x0000.

@civic crane A single 16-bit NUL suffices.

[simple ravine]

Ok, I get a feeling I should add some more tests, hmm

[civic crane]

[h 0x00] [e 0x00] [l 0x00] [l 0x00] [o 0x00] [0x00 0x00]

[worthy cape]

@civic crane Right, that's one 16-bit code unit for termination.

Say that you've got an input like:
h 0x00 e 0x00 0x00 0x3E 0x00 0x00: "he㸀"

[civic crane]

There's legitimate occurences of 0x00 0x00 in the input.
@worthy cape that's what i was not agreed.

[worthy cape]

If all you do is sweep for two consecutive zero bytes, you may have false positives in UTF-16 text.

[civic crane]

yep and that's why we strLen % 2 != 0 it

[worthy cape]

The string doesn't necessarily end at that point tho, which @simple ravine's code assumes.

[simple ravine]

It doesn't end at the first occasion of 0x000000 ?

[worthy cape]

It works out in the case of the four-byte termination scheme, but what I'm asking is if that's actually how the strings are laid out in the DAT files.

[simple ravine]

from what I gathered, thats the case

[worthy cape]

Someone ™️ should document that to reduce confusion 🙂

I've never had to care as I've just swept by code units.

[simple ravine]

I mean, I guess there could be a scenario where the last UTF-16 character ends with 0x00 and for whatever reason there's a 0x00_00 0x00_XX in that is actually in the string?

[civic crane]

yeah, when my offsets was broken i got chinese chars at end))

[worthy cape]

There can never be more than two 0x00 bytes in the middle of a string, as then either the first two or last two are a UTF-16 NUL.

[civic crane]

let end = findSequence(data, [0x00, 0x00], offset)
while ((end - offset) % 2 != 0) {
  end = findSequence(data, [0x00, 0x00], end + 1)
}

[worthy cape]

The quad-NUL seems to be a thing I guess, squinting at DAT files.

Please interpret all my communications beforehand as assuming that the termination is single, as it's in every other file format in the game 😄

[simple ravine]

got me thinking there for a while heh

[worthy cape]

Please note that other formats don't have it ^_^

[simple ravine]

I also have some safeguards if they null-value the offset or elements

public static bool IsNullValue(this int val) =>
    val is -0x1010102;

public static bool IsNullValue(this uint val) =>
    val is 0xFEFEFEFE or 0xFFFFFFFF;

public static bool IsNullValue(this long val) =>
    val is -0x101010101010102;

public static bool IsNullValue(this ulong val) =>
    val is 0xFEFEFEFEFEFEFEFE;

which reminds me, I should aggressively inline those

[worthy cape]

Note that refs may be into the bbBB landing zone, I believe.

I don't remember if it was bugs in my loader or if there's ref|string into the first 8 bytes of var-data and exactly one-past-the-end of the var-data.

It's been like a year since I tested on the whole schema.

[simple ravine]

If you can recall what file, I'll take a peek

[worthy cape]

No idea.

[simple ravine]

I am starting to consider to code-gen classes based on the dat spec

[worthy cape]

I'd love to verify and check, but so many other problems closer to heart.

[simple ravine]

I did one of those a couple of years ago, but kinda moved away from it

[worthy cape]

You've got two choices really, either something efficient around a dynamic schema, or codegen-all-the-things.

[simple ravine]

Right now, I have pretty efficient schema

[worthy cape]

I'm leaning more toward a dynamic schema myself, as I don't have much in the way of reflection and generated code is a bother to troubleshoot.

Not too far from working with shader uniforms and sqlite binds, so should be workable enough.

[simple ravine]

well, there are some nice tools around code-genning in c#

I'd code-gen in a pre-build phase, to output .cs files

but that'd be 656 cs files 😄

not sure if I want to deal with that

[worthy cape]

My primary concern is handling multiple schemas as I want to consult historical files.

Just compile into a separate assembly and no-one has to know 😉

[simple ravine]

haha

It'd be more for convenience than performance

I think performance improvements would be minimal

dotnet run -c Release -f net5.0
Loading specification... 59.2444 ms
Enumerating dat files... 19.38 ms
Reading all rows in dat file Mods.dat... 25.2549 ms
Done in 103.8793 ms

it's OK enough, especially considering it's using a quite minimal amount of RAM

all I do is load the raw bytes into memory, and use them on-demand

but yeah, just reading 2 fields from the Mods.dat in that example

would take a bit longer if I read all of them, I guess

but it doesn't allocate a bunch of crap on the heap, which is what I want to avoid

[civic crane]

what's the most heavy file, MonsterVarieties?

[simple ravine]

yeah I think so

oh wait

GrantedEffectsPerLevel

[civic crane]

even it has more rows, i think MonsterVarieties is longer to parse 🤔

[simple ravine]

Ok, let's make a little test

I'll take MonsterVarieties

    public class MonsterVarieties
    {
        public string Id { get; init; }
        public ulong MonsterTypesKey { get; init; }
        public string[] AoFiles { get; init; }
    }

something like that perhaps

as a first test

Andre E:\..\..\PoeSharp\..\..\DatRead git: new ≣ +7 ~21 -6 ! ❯❯❯ dotnet run -c Release -f net5.0
Loading specification... 58.865 ms
Enumerating dat files... 22.3693 ms
Reading all rows in dat file MonsterVarieties.dat... 23.0181 ms
Done in 104.2524 ms

[civic crane]

the point is complexity of type conversion of all columns. in above case GrantedEffectsPerLevel obviously longer I think

[simple ravine]

yeah, still have to write all methods for referencing other dat files to return that DatRow etc

but will do that today

var ret = new MonsterVarieties[c];
for (var i = 0; i < c; i++)
{
    var row = dat[i];
    var id = row.GetString("Id");
    var mtk = row.GetPrimitive<ulong>("MonsterTypesKey");
    var aoFiles = row.GetStringList("AOFiles");

    ret[i] = new MonsterVarieties
    {
        Id = id,
        MonsterTypesKey = mtk,
        AoFiles = aoFiles
    };
}

this took 23 ms though

not too bad.

not too many rows though

outputting 10 first rows just to see how it looks like

[civic crane]

referencing other dat files
nah, I mean just raw parsing (without going deeper into referenced dats')

[simple ravine]

oh well that's easy enough 🙂

just memory-marshalling bytes into whatever

for non-strings

[frank kayak]

finally found out how edge encoded secure and httponly in the past in WebCacheV01.dat...

[simple ravine]

you're going hard into reading poesessid from cookies huh

[frank kayak]

...yeah

[simple ravine]

nice 🙂

[frank kayak]

netscape is long done

[simple ravine]

netscape 😂

i guess low hanging fruit

[frank kayak]

i think i overdid it a bit

only a bit though

those bit flags werent documented on any forensic site

[simple ravine]

Chrome, Firefox, Edge, Edge Chromium

and I think you have 99% of usecases

[frank kayak]

they are easy

needed a challenge i guess

[simple ravine]

Brave, Safari, Opera

[frank kayak]

for chrome its slightly more than saltysalt peanuts

safari did already work, opera is now chrome without profiles

brave is prob also just mozilly or chrome?

old opera -> todo list

thx!

need a bit Presto

[simple ravine]

yea I think Brave is a derivate of one of them

looks Chrome-ish to me

but not sure

never cared too much, just use it on my phone

[frank kayak]

chromium based

yeah

[simple ravine]

ok, time for shower and grocery store

i'll be back to test the whole MonsterVarieties in a bit, Snos

[primal schooner]

why is https://www.pathofexile.com/api/character-window/get-items?accountName=aeonabyss&character=thefrobel_h_spectres erroring ?

[civic crane]

remove api/ ?

[primal schooner]

oh whoopse copied the leader URL over from the stash one I was working with. thanks

[worthy cape]

@primal schooner You're not specifying realm btw.

cf. https://github.com/PathOfBuildingCommunity/PathOfBuilding/blob/3f346b68211e0b6f2b4913e317247d1fdbd4a981/Classes/ImportTab.lua#L453

[primal schooner]

can someone have the same account name and character name on the Xbox as I do on the PC ?

[worthy cape]

Don't know, but PoB specifies it 🙂

[primal schooner]

this isn't being used w/ POB

but i'll keep that in mind

thanks

[worthy cape]

The website does too.

[primal schooner]

that's not in the documentation and it pulls fine w/ account and char name

if i run into an issue with it, i'll try adding realm

I just had copied over the base URL including the API part cause I was doing stuff w/ the stash in another tab of postman

[worthy cape]

Documentation as in the swagger listing? I'd trust the active user of the interface more than rather aged and empirical docs.

But hey, keep doing things differently I guess 😄

[primal schooner]

active user of what interface?

isn't this a GGG endpoint?

[worthy cape]

The website, the official consumer of the endpoints you're trying to interact with.

PoB's usage mirrors the website's usage.

[primal schooner]

there is no website, this is me doing querys in postman

[worthy cape]

....

https://www.pathofexile.com/account/view-profile/aeonabyss/characters

[primal schooner]

sorry i'm pretty new w/ this stuff, so i'm not sure what yo umean.

[worthy cape]

This is where the usage is discovered from, the character listing in the user profiles on the pathofexile website.

The endpoints exist to power this page, pretty much.

[primal schooner]

right, it's a GGG endpoint, so i don't get why you are asking about POB

which is not GGG

[grave wren]

because pob uses the endpoint in a way that works so you can just check out how they do it

alternatively use your browser check XHR requests

[primal schooner]

oh, their code is public, so you can see how they do it.

i get it

[grave wren]

no it's not public but it's on the web so you just see any requests they have to make

[primal schooner]

ok, thanks. i'll keep that in mind if using it w/o realm encounters any issues

https://www.pathofexile.com/character-window/get-items?accountName=aeonabyss&character=thefrobel_h_spectres returned the anticipated results

[wintry surge]

Thinking about refactoring my dat parser to use https://docs.rs/zerocopy/0.3.0/zerocopy/ - which would mean I do lazy evaluation of strings and anything in variable data. I think it'll simplify the parser and certainly some perf improvement, but I don't how to handle ownership and all that just because having each record maintain a copy of variable data is a terrible idea. @simple ravine how are you doing deserialization / string evaluation?

My current system with parallelism at the record level gets the same speed as yours so I'm really curious how you've been doing it

[primal schooner]

I jsut accidentially had API in the URL

[worthy cape]

@wintry surge I bit the bullet and copied my strings as I wanted to transcode to UTF-8 anyway and handle endianess when reading.

Never looked at zerocopy.

I had similar lifetime trouble with the GGPK bits, as I had a memory mapping that was to be shared between file and directory entries and the pack itself.

Ended up lifting it into an Arc, which has some overhead but as access was on a per-file basis, it was fine.

For something more granular you may need to actually deal with lifetimes, which makes it urgh.

[simple ravine]

I read bytes into string when user is requesting the column

and return it to the "user"

[wintry surge]

Yea what I'm trying to wrap my brain around with lazy evaluation is that if you lazy-evaluate at the record field (in a potential array of strings) level it needs some sort of reference to the variable data but the entire "file" should have ownership

[simple ravine]

well, in c# there's something called Lazy<T> which you can use to give it a delegate of what to do on first access. However that didn't work well, as every Lazy<T> would take 24 bytes.

[wintry surge]

Lifetimes might not be the worst, but declaring lifetimes in a proc macro sounds scary

[simple ravine]

Each column of each row

no bueno

So my Dat thing is just an intelligent holder of bytes.

[wintry surge]

Ah, I think something like that could work. I think rust has a structure that does that but I need to ensure that it's type size is consistent with the size in static data because zerocopy uses the struct size to "deserialize"

[simple ravine]

yeah, I have no idea about Rust

but zerocopy sounds dangerous, wouldnt that mean u return an address to the place with type information?

[wintry surge]

Ill start with just zerocopy to deserialize jus the static data and go from there

[primal schooner]

ladders endpoint will only ever return 15000 total entries, correct ?

[simple ravine]

if the user would manipulate that, it'd potentially blow up?

Frobes, yes

[wintry surge]

I don't think so, but if my adventure fails I'll let you know lol

[simple ravine]

The door between home office and living room has... had... a frosted glass window

[worthy cape]

Now you can See Sharp?

[simple ravine]

6 year old kid got mad when I told him to go brush his teeth

The window "blew up" in front of my hand

cant wait to gf to come home tomorrow from work trip

alright brb, gonna check if kid is sleeping

just got done cleaning up glass and blood

[grave wren]

time to buy a screen to fit into the door

[simple ravine]

yeah, no

[grave wren]

modern problems require modern solutions

[simple ravine]

more like, sturdy wood with dampening things or something

[worthy cape]

Stay safe, exile.

[grave wren]

but without glass you wont know if a murderhobo is coming into your homeoffice

[simple ravine]

lol, the only ones coming in here are the neighbors when I dont hear then knocking

[simple ravine]

Hmm, I wonder why in BaseItemTypes, There are two Sacrifice at Midnight, for an example, where one is pointing to the other.

[obtuse citrus]

look at item class