tooldev-general

[grave wren]

worked fine for me with switching

[frank kayak]

on twitchrivals it said drops was activated on mathils and it is only possible on one stream or something like that. couldnt find a button to switch. got both mtx now though so whatever 🙂

[grave wren]

weird maybe it bugged out in my favour then

[frank kayak]

just be happy when twitch drops works without problems

[simple ravine]

oh well

I have too much mtx anyways

[grave wren]

@fickle yew links to some (maybe all) awakened gems seem broken - tried it with chaos, fire and another one i dont remember: https://poe.ninja/challenge/skill-gems?name=awa

[frank kayak]

in go that would be omitempty

[frank kayak]

standard lib json finally works in tinygo! was prob this https://github.com/tinygo-org/tinygo/pull/1787

[broken cloud]

@fickle yew Also, no listing for Expedition artifacts yet?

[vapid pulsar]

Properly setup my tool for looking up Gwennen gamble outcomes.
https://gwennen.eingin.dev/

Feedback welcome

[fickle yew]

Well they largely made awakened gems irrelevant with the patch.. But sure I'll fix it 😅

Thanks for the quick fix!

[fickle yew]

Working on it. Should be able to add them today.

[grave wren]

Don't do that to my boy awakened added chaos

[fickle yew]

Fix deploying now.

[grave wren]

Thanks

[simple ravine]

@compact isle there seems to be some maps indexed that are not in the /api/trade/data/static content, fyi (in Expedition league)

[simple ravine]

I'm also getting some intermittent 502 responses.

[simple ravine]

getting there 🙂

now to figure out a good way to store this

[frank kayak]

/dev/null can store quite the amount 😛

is that tool published?

does the parsing of the items block the next fetch?

[frank kayak]

just got a letter from the police that the lawsuit against the burglar from march was temporarily dropped. dude is on the run...

i am getting the feeling that "please break in" is somewhere written on our house. 4th one so far. ...counting

[simple ravine]

yes, but not the last commit

[simple ravine]

no, it's separated

[simple ravine]

the 4th?! time to move?

[worthy cape]

Put up signage:

Intruders will be informed at length about the virtues of Go

[simple ravine]

I still have to add things such as error handling 😅 and parsing of notes

but matching item basetype/name to buyout-tags works nicely

@frank kayak https://github.com/andreandersen/PoeCurrencyIndexer

[frank kayak]

bookmarked

[frank kayak]

nah

nothing to get here or youll have to find it first...

😛

[simple ravine]

time to add lazer turrets?

[frank kayak]

that dude was in a state of shock after i appeared. think that was enough. then he got hunted down by the neighbors kids

they are already adult. time files...

[simple ravine]

and the other 3 times

[frank kayak]

they got away but prob children in those cases

[frank kayak]

go > rust

fite me

[simple ravine]

https://giphy.com/gifs/colbertlateshow-pUSfiW4GNU0gg

[frank kayak]

chicken from yesterday got released into the hideout

[simple ravine]

ok time to do some chores, then this code-base needs some refactoring, it's a mess

[frank kayak]

wanna see mine? 🙂 - can invite you on gitlab. yo do the clean up?

zao already had a look

[simple ravine]

everything has a price 😄

[frank kayak]

as the price ill publish it 🙂

[simple ravine]

I'll clean it up real nice. Can't guarantee it's golang afterwards though

[frank kayak]

have no vigor to play this league. barely arrived in act 3... - at least coding is fun

[simple ravine]

I'm in A1. Got the medicine chest.

[frank kayak]

theres too much even in the fugues you cant get rid of all of it

gz!

[simple ravine]

I was really hyped about the nerfs and stuff, then realized they're not that nerfy anyways

[frank kayak]

with that corrupted oil from last league i could get the nth charge for my flicker discharge... let me check what n was

[simple ravine]

League mechanic from the surface doesn't seem to have any end-game goals, so meh.. no reason to go there

[frank kayak]

im fine with those. prob more coming - im not fine with the state of "gambling" i want "crafting"

acquisition of crafting resources can be rng - totally fine with me. crafting process - nono not fine

[simple ravine]

I don't want to grind several hours to experience something that's mildly challenging

[frank kayak]

n = 22
want a 23rd charge (end) on the boots

[simple ravine]

thats a lot of charges

[frank kayak]

never enough 😄

[simple ravine]

you clean maps in < 10 seconds? 😄

[frank kayak]

but the aoe shouldnt scale over the flicker range was my prob before i got alt qual flicker

great mapper and very old prob 5 years or older - with harvest i could make him take a few hits

but not great for bossing

old blobs (volatiles) bears and reflect mobs were fun

[simple ravine]

yea, flicker isn't really optimized for bossing

never played it. can you do something like 2x 6-link and have an alternative bossing setup?

[frank kayak]

not flicker the charge gen for discharge did fall down

[simple ravine]

like I used to do with Tornado Shot + Barrage

[frank kayak]

dont want to have a bossing setup

im lazy when playing

if it doesnt work ill do it until it does work

i think ill start ripping out the internals of my tool...

[grave wren]

Is there a good go hands on page like automate the boring stuff in python?

[frank kayak]

did i just read "Go"??

what do you want?

https://tour.golang.org/ fastest intro

learning go doesnt take long.

mastering might take a bit

[worthy cape]

Scrubbing it from your mind takes longer.

[frank kayak]

😛

just dont

coding in go is fun

in other languages i dont feel the fun

perhaps bash and sed but im weird...

[grave wren]

dunno the tour is a bit too low level for me, i'd like something like smaller projects you can do

or rather things with a purpose like web crawling, ...

[frank kayak]

colly is a scraping lib

[grave wren]

https://automatetheboringstuff.com/2e/chapter12/

[frank kayak]

you could use the libraries llying below (antchx xpath stuff) to do it yourself with standard http lib

[grave wren]

compare it with this :p

[frank kayak]

ah

[grave wren]

just hands on intro

[simple ravine]

try something like Haskell or F#

[frank kayak]

no requests needed standard lib is good

colly for scraping

let me check for that selenium replacement...

[grave wren]

i dont mean what to use more like a hands on intro besides a "this is how the language works" dont know how to better explain it 😄

[frank kayak]

chromedp automates your existing chrome browser no need for additional stuff like with selenium but we also have a dozen selenium libs in go

hmm

[worthy cape]

Kind of like the Rosetta Stone projects out there, but a bit larger in scale?

[grave wren]

i mean doesnt even have to be a whole project but just a blueprint on how to do x with go

[frank kayak]

scraping: http://go-colly.org/docs/examples/basic/

[grave wren]

always helps me to see snippets work in a specirfic context

[south gorge]

Hello,
I have a question about OAuth2. About a year ago I developed https://github.com/Essyer/PoETiS. Just today I learned that the way I obscured my tool to look like some browser sending requests is something I should change ASAP. Requests are limited to 1 per second and there is only one person I know using this tool, still I want to fix it. On pinned docs page, section Authorisation I can see that "...any distributed application (one that runs on a user's machine) will need to be backed by a secure server in order to keep your application credentials secret.". Does that mean that I need to rent some server to store the client_secret? I tried to find an example use of OAuth2 and in exilence code I cannot find anything about client_secret, only client_id is used in line 107: https://github.com/viktorgullmark/exilence-next/blob/35661c636604d7189cb9b11541ead727f908ce74/ExilenceNextApp/src/store/accountStore.ts#L92

[frank kayak]

browser automation examples: https://github.com/chromedp/examples#available-examples

as i understand it to auth with the poe api site with an oauth2 token you need a server. for my client tool ill continue using poesessid

[worthy cape]

Regardless, a distinct user-agent with contact info is good.

[frank kayak]

so a chrome browser faking a safari browser faking a firefox browser is not ok?

hmm

[south gorge]

Thank you! I will add my contact info, but what do you mean it has to be distinct? I obscured it because sometimes I was not getting response, I don't remember what code it was.

[frank kayak]

relying on the other party to truthfully report sucks - ip based controls makes more sense to me

[worthy cape]

Like toolname, author, version.

[frank kayak]

"github.com/Essyer/PoETiS" in the user agent is prob enough

[worthy cape]

It ensures that you're not routinely blocked because you use a stock requests or other library user-agent string, and it makes it so that if a version of your tool misbehaves, they can get hold of you and they can block specific versions.

[south gorge]

Thank you, now I get it 😀

[frank kayak]

in most cases i fake the most common browser user agent when scraping (not for poe as not required)

[gilded tree]

https://gophercises.com like that?

[grave wren]

looks cool, thanks

[frank kayak]

🍕 now im full...

[simple ravine]

that reminds me, time to order something to eat

the new IntelliCode in Visual Studio 2022 is pretty slick

[frank kayak]

im fine with VSCodium atm big upgrade over SciTE

the dark color mode i created for scite was aweful

[simple ravine]

I was thinking about the AI powered code completion suggestions

[gilded tree]

I tried CoPilot quite heavily, it's really amazing... sometimes. But if it autocompletes the whole function correctly it's an awesome feeling.

[simple ravine]

I'm still in waiting

[frank kayak]

ai powered sounds scary to me

i prefer getting suggested those 100 line long base64 strings in vs code

not sure why that happens though...

did anyone try nuitka the last year? did the static compilation get better?

[frank kayak]

seems more mature now - have nothing to test with though

[simple ravine]

The one in VS22 does not send the code continuously to some server AFAIK

but it's not CoPilot

[simple ravine]

that's some decimals, hmm

wonder if I should truncate that a bit

or store the fractions somehow (or whatever the english term is)

[worthy cape]

Most fractions are not exactly representable in floating point reals, so keeping them in a normalized form probably helps.

You can always compute a real if you want to do serious computation from that.

[frank kayak]

hmm my xargs sed -i accidentally hit the .git folder pointing my repo at a different repo... nice

[civic crane]

Community feedback https://www.reddit.com/r/pathofexile/comments/ougsnd/comment/h72r7am/ @velvet fog

[velvet fog]

fixed the audio issue

[frank kayak]

damn ad

[frank kayak]

was able to separate the ui code from the rest of my code base...

now a few more things to rip out

perhaps i have it in an acceptable state earlier than i thought 🙂

[frank kayak]

https://old.reddit.com/r/programmingcirclejerk/comments/ooc3fs/rewrote_golang_project_in_rust_its_4x_times/h5y37r2/?context=3 🦀 @worthy cape

[worthy cape]

Naive transliteration performs poorly, news at 11 😄

[frank kayak]

never had sushi

[frank kayak]

hmm - is getting the options from the trade site as annoying as this:

curl -sL https://web.poecdn.com/js/trade.2e416d04dec4fbeaf67a0b312701255287ff72b5.js | sed 's|.*define("PoE/Trade/Data/Static"||;s|}),.*||;s|.*;return{|{|;s|e\.translate(\("[^"]*"\))|\1|g;s|:!0|:"NOT_NULL"|g;s|\([^[:alpha:]_]\)\([[:alpha:]_]\+\):|\1"\2":|g' | jq -Cr .

?

[frank kayak]

looks like this when forced into json format: https://pastebin.com/XKTaGXEQ

[rapid pagoda]

Does anyone know offhand how the mapping works from symbolic artwork paths like Art/2DArt/UIImages/InGame/NPCWindow/Portraits/Trialmaster to DDS file paths?

[worthy cape]

Art/UIImages1.txt is a sprite sheet index.

Maps from string to file and rect.

"Art/2DArt/UIImages/InGame/NPCWindow/Portraits/Trialmaster" "Art/Textures/Interface/2D/2DArt_UIImages_InGame_NPCWindow_Portraits_4.dds" 859 758 1708 1507

[rapid pagoda]

Aha. There we go

There's a few more of those actually. Most interesting one is the div card index, although it looks as though all of those got split out of their sprite sheets at some point

Art/UIDivinationImages.txt
Art/UIImages1.txt
Art/UIPS4.txt
Art/UIShopImages.txt
Art/UIXbox.txt

[rapid pagoda]

Finally got around to working on the stacked-FS loader. Got pogo cat working with a Steam install; implementing ReadDir for bundles is going to be a bitch and a half though

[rapid pagoda]

@hazy fog accurate.

[split fjord]

how does pypoe remember config after I cleaned the repository? there must be something stored in a temporary folder somewhere but I can't locate it

[split fjord]

found it, it's from qt and it uses %APPDATA%\PyPoE\exporter.conf

[split fjord]

@woeful sphinx is pypoe_ui working?
it seems not to do anything after sorting the file list (HEAD -> dev)

[woeful sphinx]

Nope, never got updated to support bundles

[split fjord]

damn

nor the exporter I presume?

[velvet fog]

you need to export .dat files from other tools if no bundle support

[split fjord]

I see

[woeful sphinx]

The exporter might work. Not sure how far exactly pypoe's bundle support goes.

[split fjord]

I thought you were the maintainer of it hence why I asked

and it has commits from 3 days ago

[woeful sphinx]

I'm only maintaining what RePoE needs, don't really know much about its other features.

[split fjord]

Ah okay then

all this stems from a few days ago I noticed some stuff on the wiki was out of date, then I noticed omega's export bot wasn't doing stuff, and now I've spent two days ferriting around the PyPoE only just tonight finding out it was discontinued, and now I've just figured out yours is too

worst part was the four or so hours I spent modifying pyside to build on 3.9, finally giving up and downgrading to 3.7 zzz

[woeful sphinx]

I'd be open to PRs on other areas too. But as I said, don't know enough about PyPoE to maintain everything myself and don't feel like spending a bunch of time to get to that level either.

[split fjord]

yea, I feel ya - I don't personally want to learn python as I come from the world of c and I find it very slow and hate that it lacks braces heh

[frank kayak]

not a fan of stuff blowing up in it when the code base gets larger and something tries to digest a type it shouldnt

and hate the bad deployability a lot. should be as easy as go rust crystal and end up in an independent executable

[frank kayak]

cant find the commit. not yet pushed?

[rapid pagoda]

not pushed, I'm still working on the the ggpk side

literally just got file loading working and I'm about to crash for the night

[frank kayak]

ok looking forward to it

[split fjord]

@woeful sphinx do you have an old PassiveSkillGraph.psg at version <=3.14?

or does anybody else for that matter

[velvet fog]

3.14.3.2

[split fjord]

tyvm

damn its the new version, any idea when they changed the format?

or has someone successfully parsed the new data?

[velvet fog]

they added 1 byte after group_count.y in 3.10

[split fjord]

alright I'll see if that's enough thanks

wait are you sure? because the root_length is in the order of 200million, if shouldn't be that large

[velvet fog]

oh, 1 byte after version too

[split fjord]

also what's group_count.y, is that a reference to some other project?

[velvet fog]

nope, my own private project ..hmm..

3.9.3.3

do you have hex editor or 010 editor?

[split fjord]

yea

I use vim for this sort of stuff usually

vimdiff with xxd

cheers

[velvet fog]

can you read this .bt file?

[split fjord]

it looks like c to me so yes

or maybe something more like c++

[worthy cape]

I'm still amazed that my one-week file extractor from back when bundles were introduced is still robust enough with a few minor fixes to be used in production.

[split fjord]

hmm?

I know you authored ooz, but not sure what that project is

[worthy cape]

In that repo is bun_extract_file, which lists and extracts files from the game bundles.

[split fjord]

also why on earth does the new LibGGPK2 require oo2core_8_win64.dll

[worthy cape]

(I didn't make ooz itself, just patched and wrapped it for our tool use)

[split fjord]

ah ok

[worthy cape]

There's two ways to extract Oodle-compressed data. Either via official RAD Game Tools (now Epic Game Tools) DLLs, or with the reverse-engineered reimplementation "ooz".

[split fjord]

right is this what oo2core_8_win64.dll is?

[worthy cape]

Yep, you typically find it bundled with games like Warframe.

PoE also has a decompressor, but it's statically linked into the executable and of no use.

[split fjord]

I mean, as far as re goes, that's probably no more difficult to reverse than the RAD ones

[worthy cape]

The bundle structure introduced after the tech patch introduced bulk block-compressed data files, and a binary index with file records noting what slices of the decompressed data files to cut out, and data with which to generate the directory and filename paths of file records.

Most old tooling could be adapted from the old-style GGPK reading to reading bundles from GGPKs or loose files, but PyPoE is not quite all there as the old structure kind of touched most of the tool.

PoB's exporter settled for extracting files to a temporary directory and working directly against them.

[split fjord]

yeah this temp directory export is incredibly slow for testing code changes lol

[worthy cape]

You can read up on some of the stuff at https://github.com/poe-tool-dev/ggpk.discussion/wiki/Bundle-scheme

[split fjord]

is it at least going to be in ram because I've extracted it so many times I worried about wearing out my m.2 :S

[worthy cape]

An efficient exporter would take the set of files you want, sort them by bundle ID, determine which blocks it needs to decompress and slice file data out of the resulting blocks.

[split fjord]

yeah, it's insane that it parses the entire thing every time

the only reason it doesn't take half an hour is the speed of this disc too, 3.5GB/sec reads/writes

[worthy cape]

Which tool are you talking about now?

[split fjord]

pypoe_exporter

[worthy cape]

When working with the GGPK VFS, you kind of have to do a full scan of it to build up your directory structure. You can scan linearly and collect chunks as you go and reconstruct the hierarchy from that, or you can chase pointers all around your disk, which would let you only touch a subset of the entries, but often costs more than a linear sweep.

My tooling that still cares about GGPKs do the linear sweep to find all chunks, and then link them up in memory.

[split fjord]

is there not some index for it all?

[worthy cape]

But yeah, you could easily have like 16s of startup time.

GGPK is node-oriented in chunks. Directory entries index other directory/file entries by filename hash, and file entries store the data in-line in the chunk directly after the header.

In order to read a filename/size/metadata for a directory entry, you need to follow the offset to the file/directory entry it refers to.

[split fjord]

crazy

[worthy cape]

The tech patch in September 2020 introduced the new bundle structure, which has O(1) resolution of file entries by path hash.

The bundles are either stored directly on disk (with Steam) or in a traditional GGPK (with the Standalone).

The GGPK scaling is "fine" as it cuts the file count from 500k-700k to somewhere around 50-60k; it especially gets rid of all the super tiny files and jams them into bundles.

[civic crane]

bricked it 😄

[outer oak]

Does anyone know where i can get all the map art ?

[worthy cape]

The sigils engraved in map item 2D art?

[outer oak]

ye

i wanted to post a pic here but i cant for some reason

[worthy cape]

They're in images somewhere, you have to tint and composite them yourself with the base map item, or fetch them via the CDN, but that's harder now.

[civic crane]

or use poe.ninja https://poe.ninja/api/data/ItemOverview?league=Expedition&type=Map&language=en
Warning: some maps have guard image on it insteda (because poe.ninja doesn't filter based on implicit 😄 )

[outer oak]

yup that works ty

[civic crane]

there is also what poemap uses https://api.poemap.live:8443/api/series/16 but these urls will break in 3.16

[primal schooner]

are there any public poe.ninja build API endpoints ?

or is it all just econ

[frank drift]

https://poe.ninja/swagger/index.html never messed with other parts of the API but looks like there's some build stuff

[primal schooner]

ah yes, ok thanks.

[rapid pagoda]

progress:

% ./pogo ls -l steam:
D              ./
D              Art/
D              Art/
D              Audio/
D              Bundles2/
D              CachedHLSLShaders/
D              Data/
D              FMOD/
D              Metadata/
F     42783472 PathOfExileSteam.exe
F       535792 PathOfExile_x64Steam.exe
D              Redist/
D              ShaderCacheD3D11.packed/
D              ShaderCacheVulkan.packed/
D              Shaders/
F       305152 bink2w32.dll
F       384000 bink2w64.dll
F      4173928 d3dcomp_47_x64.dll
F      3466856 d3dcomp_47_x86.dll
F      4173928 d3dcompiler_47.dll
F     19695104 dxcompiler.dll
F      1496576 fmod.dll
F      1795584 fmod64.dll
F      1099264 fmodstudio.dll
F      1435136 fmodstudio64.dll
D              minimap/
F       237856 steam_api.dll
F       262944 steam_api64.dll

yes, Art/ shows up twice because it exists both in the filesystem and in bundles, and yes, pogo ls -l steam:Art/ works correctly 🙂

[worthy cape]

poggo \o/

[rapid pagoda]

pushed now \o/

[simple ravine]

noice

@rapid pagoda steam = "extracted ggpk"?

[rapid pagoda]

it's intended to be used on a Steam install directory, but yes, you could also use it on the contents of an extracted GGPK

well, any newer one at least -- it always enables bundles so it'd fail if you pointed it at an old GGPK directory

[velvet berry]

Heya, wanted to ask, trade helpers like (https://github.com/lemasato/POE-Trades-Companion) utilize only client.txt right, no memory reading or OCR usually? I am thinking of making one that works on linux and has a small binary.

[worthy cape]

Pretty much universally just client.txt; memory reading is super illegal and OCR is hard.

[velvet berry]

That sounds perfect, now just to get time out of work and I can give it a shot. Thanks!

[frank drift]

GGG also has restrictive OCR rules

makes it even harder

[velvet berry]

Suppose to prevent botting, makes sense. Well, worst case scenario it means I won't get notifications from our fellow Korean players.

[rapid pagoda]

https://github.com/oriath-net/pogo/commit/82bb9a7d 🎉

[rapid pagoda]

Out of the 693 data files in 3.15:

• 380 are parsed successfully and identically as dat and dat64
• 18 succeed as dat and dat64, but the output differs (usually because an array has the wrong type)
• 100 fail under dat (format needs to be updated for 3.15)
• 128 fail under dat64 (some fields have the wrong types)
• 67 have no format definition at all yet

[velvet fog]

    const EXT_CONFIG = [
        'dat'=>['REF_WIDTH'=>32, 'STR_ENCODE'=>'UCS-2LE'],
        'dat64'=>['REF_WIDTH'=>64, 'STR_ENCODE'=>'UCS-2LE'],
        'datl'=>['REF_WIDTH'=>32, 'STR_ENCODE'=>'UCS-4LE'],
        'datl64'=>['REF_WIDTH'=>64, 'STR_ENCODE'=>'UCS-4LE'],
    ];

[rapid pagoda]

same conclusion, yeah https://github.com/oriath-net/pogo/blob/main/dat/width.go

[naive gust]

where could I go for potential help with pob fork

like with an error message constantly re apearing

[worthy cape]

https://github.com/PathOfBuildingCommunity/PathOfBuilding/issues

Make sure to look for duplicates like https://github.com/PathOfBuildingCommunity/PathOfBuilding/issues/2945

[velvet fog]

If i want to integrate a markdown or wiki like editor, any suggestions?

[rapid pagoda]

What part of the solution are you looking for? https://ckeditor.com/ has a great rich text editing frontend that can be configured to output Markdown; https://github.com/s9e/TextFormatter is a decent PHP Markdown renderer

[velvet fog]

https://pandao.github.io/editor.md/en.html this looks great

[naive gust]

that is exactly my error, ty

[frank kayak]

had some issues with PCall while trying to port. hope its unrelated

prob an issue from a called func

[worthy cape]

Exciting!

[hazy fiber]

"dded b|fien|phec|sadi|prae|occ|ge ax|or p|us p"
"stone r|her b|ua a|silk g|erer b|turq|uered g"
"ezomyte s|jud|hon kite|cob|ader p|gav|sinn|vir"
"sim|zod|ony t|udg|wo-p|amet|citr|al am|on j|onyx"```
dont those 4 cover pretty much everything?

[frank kayak]

just buy all. then you wont miss out

try .*

[simple ravine]

what are those? that looks like weird regex to me

[civic crane]

gwen search

[simple ravine]

why are some OR-separated and some space-separated?

confused 😄

[worthy cape]

PoE's filter box is a lovely mixture of heresy and regex.

[civic crane]

with quotes space doesn't mean OR

[simple ravine]

ok, if you'd like, out of curiosity the "dded b|fien|phec|sadi..." what is it trying to find there?

if it's added, why not just type added?

[hazy fiber]

"dded b" is Studded Belt for rylsatha

[civic crane]

input has char limit, descriptive regex = more buttons to click 😅

[simple ravine]

ah

[hazy fiber]

dont worry, its been absolute trash for me

only use atm is rolling for tabula imo

[simple ravine]

I'm lvl 8 in A1 😄

[hazy fiber]

yeah its ok for those

I used em for i84 bases day 1, which was nice untill other people caught on

[frank drift]

I've gotten a couple 1ex uniques, but mainly doing it to get the challenge over with

[frank kayak]

I reached level 28 - league is that good 🙂

[fickle yew]

This is the best place. Added to my long list of pob adjustments 🙂

[rapid pagoda]

working on rendering NPCConversations 😀

[vapid pulsar]

Has anyone made a tool that generate poe trade site search urls from a POB?

[worthy cape]

There was an issue this spring where a user wanted a way to build a shopping list or cost estimate for a whole build, typically ones from guides or creators: https://github.com/PathOfBuildingCommunity/PathOfBuilding/issues/2230

[rapid pagoda]

More fun new pogo stuff:

% ./pogo ls --json Content-3.15.0.ggpk:
{"name":".","isdir":true,"size":0,"source":"union"}
{"name":"Art","isdir":true,"size":0,"source":"union"}
{"name":"Audio","isdir":true,"size":0,"source":"GGPK","sha256":"45b250455f3c3211b4ed32eea146f6699202b98553b1e9482fa01d35cf34fd4c"}
{"name":"Bundles","isdir":true,"size":0,"source":"GGPK","sha256":"c294d80c56a2be79ce1a22c7b8aaccd14a7b71e96857a632849fe8a9734c9471"}
{"name":"Bundles2","isdir":true,"size":0,"source":"GGPK","sha256":"ef5d753e42061ce2865ca23b0fc033913cbfd32e1326701687396681cd049a0b"}
{"name":"CachedHLSLShaders","isdir":true,"size":0,"source":"bundle dir"}
{"name":"Data","isdir":true,"size":0,"source":"bundle dir"}
{"name":"FMOD","isdir":true,"size":0,"source":"GGPK","sha256":"de4b4b13d9747553afdf9c0a6e7556b59d01fbd23fd877d61b827a7cabd90275"}
{"name":"Metadata","isdir":true,"size":0,"source":"bundle dir"}
{"name":"ShaderCacheD3D11.packed","isdir":true,"size":0,"source":"GGPK","sha256":"9a105efa9f4ebdcfd0849c4259d7ac07deaba8e9ad0d6f99ed6c1a343d2e2fd3"}
{"name":"ShaderCacheVulkan.packed","isdir":true,"size":0,"source":"GGPK","sha256":"560eaa3aa11f423bb9bb3f3a0acdbda4585ded5c9a40b0ec016662893f5948b4"}
{"name":"Shaders","isdir":true,"size":0,"source":"bundle dir"}
{"name":"minimap","isdir":true,"size":0,"source":"bundle dir"}

@worthy cape Remember how you said you were building a content-addressed index of all files you'd seen in GGPKs? I'm kind of coming around to doing that, except backwards and just for audio for now 🙂

[worthy cape]

If you had IPv6 you could right now accidentally access my internal dev web server hosting all(-ish) unique files ever shipped 😄

Got to love when your VMs see route advertisements and generate temporary v6 addresses.

[rapid pagoda]

@earnest radish what are the numbers?

THE NUMBERS, WHAT DO THEY MEAN

[frank drift]

can use the list of ultimatum req uniques maybe? were all drop uniques included in that set?

[rapid pagoda]

also, ultimatums didn't drop common uniques

(thankfully)

[frank drift]

sounds like that dataset could in fact work then

[rapid pagoda]

@earnest radish two-stones should probably be split into the three subtypes, annoying that they all share a name

oh, but it's for regex, nm

[frank drift]

if you're going for the challenge, it's more a question of "what's most worth gambling for"

could eventually take into account the unique tier, ie presumably how likely it is to hit relative to other items

could have an option for a synthetic chaos value that takes into account tier

like headhunter isn't actually 8000c value, it's closer to 0 once tier is taken into account

[worthy cape]

That hotfix is in place btw.

[frank drift]

even a ballpark would be useful

people complain about not getting anything but maybe they're going for t0/t1 all the time

[broken cloud]

Hmm "ss si" doesn't do it?

Ah wait, it's just silk gloves you want

I felt much the same way when Cemetery and Vaal Temple were added to the list of ambiguous area names this league (they're expedition areas now too)

[frank kayak]

does "[ ]jew" work?

or posix character classes? "[[:space:]]jew"

or \s

[frank drift]

UX thing, you should have a "search" button after "minimum chaos value" field if user needs to submit the form rather than updating on the fly after something is typed

[brittle oasis]

Does anyone have the empty inventory icon by any chance

[worthy cape]

What part of the UI of you mean?

[brittle oasis]

Just the image (png, dds, whatever) for the empty inventory slot

[worthy cape]

"Art/2DArt/UIImages/InGame/4K/InventorySquare" "Art/Textures/Interface/2D/2DArt_UIImages_InGame_4K_12.dds" 812 135 916 239

[brittle oasis]

Is there a cdn that has all the files online or where do I get that from?

I havent extracted the ggpk yet but I guess I might as well, thanks for letting me know where it is

[worthy cape]

Art/UIImages1.txt maps from logical name to sprite sheet and the rectangle containing the art.

Some day I should make or unrot the tool that slices those out.

[brittle oasis]

Is PyPoe the preferred tool to extract the ggpk?

[worthy cape]

I use my own, but I'm biased 😄

~~https://i.imgur.com/EsA4HHE.png~~ https://i.imgur.com/W6NzrQe.png

[worthy cape]

Turns out that if you put two identical files in an album and delete the wrong one, the link breaks 😄

[rapid pagoda]

⬛

[worthy cape]

@rapid pagoda

I wonder how many toy services I'm technically exposing to the interwebs <_<

[rapid pagoda]

Metadata/Terrain/Doodads/Act5/Theopolis/Sanctuary/Sanctuary_skullCupcake_02.aoc ? 🧁

[worthy cape]

I have questions about the mental state of some GGG artists 😉

I have no idea which file this is, I picked a hash at random, heh.

[frank drift]

ah the PoE item window search field is max 50 chars

that's pretty punishing

really have to optimize regex

[rapid pagoda]

I mean, these are the same folks that made the wall eyes in Belly of the Beast follow you. And created the model for the "Handmaiden" in A10 Oriath.

[rapid pagoda]

showing off analysis tools some more 🙂

by tracking offsets, I can detect missing string/array fields and arrays with incorrect types 🙂

[frank kayak]

no content visible - pls fix

[oak estuary]

Turn yo gamma up

[frank kayak]

hmm...

gimp has to many toggles/thingies for me to mess up :/

art improved:

is this really used for the inventory?

[worthy cape]

It's the InventorySquare, has the wee little borders and the subtle background.

You can see it subtly in the screenshot earlier: https://cdn.discordapp.com/attachments/175290321695932416/872817570988843028/unknown.png

Game may be doing some linear/gamma remapping in the UI renderer, of course, don't have any resources on that.

Interpretation of 8bpc images is notoriously ambigious.

[frank kayak]

nope not seeing it. but interesting

[worthy cape]

Depending on screen technology and viewing angle, it can be quite noticeable in-game.

[rapid pagoda]

adjusted the levels in that image, and…

[frank kayak]

wonder how and why one arrives at the structure

[rapid pagoda]

In the end, we over-recorded in some areas, so dataminers will have a field day extracting snippets for item filters from unused dialogue.
-- https://www.pathofexile.com/forum/view-thread/3159966

I feel extremely called out right now. 🤣

[rapid pagoda]

Hmm. I think some of the Betrayal lines which are conditional on upgrades (i.e. Syndicate items) are referencing the wrong items.

For example, Vagan has the line:

A bit of elemental protection should change the balance of this struggle.
But it's triggered when he has the upgrade Tau Poh's Toes, which grants cast speed, not elemental resistance. There are a couple of others that don't seem quite right either, like several Betrayal members implying that Akanilim's Hide grants fire damage, when it actually grants fire resistance.

[swift beacon]

I'm surprised he didn't mention Gwennen's dialogue. It's the only case I know of where a dialogue option changes after the first time you listen to it. I wouldn't have noticed if I hadn't datamined all the dialogue.

[velvet fog]

https://poedb.tw/us/Gwennen#GwennenDirectory

[rapid pagoda]

Gwennen isn't the only one. I believe Tujen will do that as well

Yes. The first time you ask him about "The King", he'll reply:

Nice weather we're having, ain't it?

If you ask again:

Wraeclast is a touch dangerous. I imagine you've got quite the story to tell.

And afterwards:

Do you not understand social cues? Stop asking!

Aside: this is actually an amazing test case and I'm now much more certain about how quest flags interact with NPC talk.

[oak estuary]

That's awesome. Sounds like I should do the All Ears achievement before more main NPCs come out with that feature 😛

[rapid pagoda]

For posterity: the three arrays in NPCTalk currently named Data0, Data1, and Data2 appear to represent "quest flags required for this dialogue to appear", "quest flags which make this dialogue disappear", and "quest flags set when you hear this dialogue".

Incidentally, this means that some of the internals for certain quests are incredibly messy. For example, Victario's Secrets (the one with the three busts) has way more NPCTalk entries than you'd expect because it needs six different cases for "you've got bust 1, you've got bust 2, you've got busts 1 and 2…"

This would also explain why Yeena doesn't have any special responses for partially collecting the fireflies. The number of talk entries required would be ridiculous

[frank drift]

factorials are a wonderful thing

[rapid pagoda]

It's just exponential, not factorial, but close enough 🙂

[frank drift]

a few seconds after pressing button, chrome

[rapid pagoda]

… wait, .py? so many questions

[frank drift]

enter "10" in chaos field, press enter, wait for list to change, click "generate string"

didn't realize you want user to click 30 checkboxes, maybe a "all" control at the top?

also this is interesting

asenath being that much

worked, regex string packing could maybe be slightly optimized, an additional item could fit in both of the first 2 although 3 total still necessary

[frank drift]

I don't think prismatic ring can generate anyway, can it?

tried all 4c non-jewellry bases, almost fits into 4 lines

not bad

can go through it quickly with keyboard macros

[frank drift]

haven't had a chance to test it yet

[worthy cape]

Not sure if automated removal or a horde of gnomes , good job regardless from the mod peeps.

[frank drift]

I see Runic Crown snuck on the list, I'd think either the base type wouldn't generate or the Cadigan's Crown would be a boss drop?

same with other expedition base types actually

I see Maw of Mischief too, which I think is div card only?

heh

[worthy cape]

Ooh, adjacent string concatenation? Didn't even realize that would occur there.

Best feature of all the languages that have it 😄

Same postcode.

[frank drift]

I'd suggest 1-based indexing here to avoid confusion 🙂

something in "jud|cul|aet|dis|ava|mpl|nne|eig|o-p|c l|75|ji" is matching nearly every item, I'll narrow it down

it's "mpl"

for 'simple robe'

"implicit modifier"

yep

[worthy cape]

[frank drift]

you know someone's going to do a level 75 window

heh

[worthy cape]

Your feature request has now been added to the GGG Redmine.

Oh. Can't have that then.

[frank drift]

I hand fixed it in my macro

now that I have the macros set up, might not bug test the tool so much any more

[worthy cape]

Totally not a buttcoin miner.

[grave wren]

slothcoin miner please

[worthy cape]

Are those based on proof-of-not-work?

[grave wren]

ironically still proof of work

yours would be more fitting

[simple ravine]

221 MiB in memory to load the files and directories into memory from _.index.bin

that's a bit..

[worthy cape]

Does that include fully rehydrating the full paths?

[simple ravine]

yeah

made into a tree structure (directories, sub directories, files) and their names, sizes and hash

152 MiB in managed heap size

[worthy cape]

I'm running my ingest tool in an 8 GiB VM. So easy to run out of memory 😄

[simple ravine]

would be interesting to compare with pogo and other tools, if it's comparable

it's ok-ish fast I guess.. 0.8 seconds (with a hot file/os cache)

dotnet run -c Release
00:00:00.7916122

from the Content.ggpk

[worthy cape]

Legacy GGPK ingest is sadness, especially over NFS.

[simple ravine]

I added lazy loading of subfolders in the GGPK, because otherwise it'd take too much time

but it's not feasible with the bundle stuff

[worthy cape]

real    0m54.632s
user    0m1.927s
sys     0m3.707s
Content-923048490654768032.ggpk

real    0m43.893s
user    0m1.972s
sys     0m3.793s
Content-937891701133792760.ggpk

real    0m31.250s
user    0m1.042s
sys     0m2.252s
Content-986068321601222572.ggpk

I do the cool thing and linearly scan for nodes up-front, reconstituting them into a tree later as I often need the whole file anyway.

Slurps at disk speed pretty much, it's just a fair bit of data to jog through.

This weekend I'm going to do a limited ingest of just DAT files and see if I can get the file search service designed and running.

[simple ravine]

I just find the root, and return that, then memoize/lazy-load

[worthy cape]

I found that when doing that, I was dying over time from a bajillion page faults when traversing everything, especially over network or from spinning HDD.

A full sweep up-front, despite costing a bit, was way superior if you touched some or most files.

[simple ravine]

var ggpk = new GgpkFileSystem(@"D:\Games\Path of Exile\Content.ggpk");
var bundleDir = ggpk.Directories["Bundles2"];
var bundleIndex = new BundleIndex(bundleDir);

var metaDataDir = bundleIndex.Root.Directories["Metadata"];
var someFile = metaDataDir.Files["filtered_words_client.txt"];

= 0.8 seconds

const string path = @"c:\noindex\ggpk\Bundles2\";
using var indexFs = File.OpenRead($"{path}_.index.bin");
var bundleIndex = new BundleIndex(new DiskDirectory(path));
var metaDataDir = bundleIndex.Root.Directories["Metadata"];
var someFile = metaDataDir.Files["filtered_words_client.txt"];

= 0.8 seconds

I'm fairly happy that the GGPK virtual file thing works out fairly fast. I think I can make it even faster though with the new RandomAccess thing, which allows me to skip using FileStream

[worthy cape]

If one was cool, one'd do something hybrid that sweeps in the background but can still service ad-hoc requests early.

[simple ravine]

I made an exporter that does discovery and exporting with multiple threads

works well on SSDs

[worthy cape]

WTB 11T of SSD.

Offer 2c, u there?

I look forward to see what kind of perf I can get from nginx with a big-ass dataset.

[simple ravine]

AV killing perf like mad tho

I really want to optimize this code though:
https://gist.github.com/andreandersen/425159dc57590a65064bb6edcc7f63e4

[simple ravine]

trying to implement bundle decompress of the other bundle files, but a bit lost.

it looks like all files have the same kind of header... the file_info->file_offset and file_size, how are those used in relation to the bundle file?

[worthy cape]

You hash the filename and find the file record that has that hash.

Use that file record's bundle ID and offset/size to decompress the right range of the right bundle.

Citing the dev wiki:

For example the file Art/UIDivinationImages.txt will become art/uidivinationimages.txt++ and resulting hash 0x574cc9062dcda786. This can be used for looking up a file to it's corresponding section of a bundle.

(https://github.com/PoE-Tool-Dev/ggpk.discussion/wiki/Bundle-scheme)

Hashing being FNV1a of the lowercase absolute path of the file, appended with ++, while directories keep their case but also get the ++ appended.

[rapid pagoda]

… wait there are directories in bundles? I was synthesizing that from the file list 👀

[simple ravine]

yeah I got that far

[worthy cape]

The directories are all in the encoded dir scheme.

[simple ravine]

I read the filename, file_size and file_offset

[worthy cape]

You may hash a directory name to find its entry in that section cheaply.

File records and the bundles are all just for file data.

[rapid pagoda]

Oh wow. Time for me to reimplement bundlefs again 😆

[simple ravine]

Right, I am trying to extract a specific file from a specific bundle, i.e. Metadata/filtered_words_client.txt

so I found the bundle.bin. file that it correlates to, but then I am uncertain how to extract the specific content

[worthy cape]

Given a path to a file you hash it to find the file record in the index, you open the bundle file given its bundle id, you decompress the blocks that cover the file data and slice it out.

The bundle file has a series of compressed blocks. Each of those decompress to a specific amount, 256k

[simple ravine]

ok so the file_size and file_offset correlates to compressed and not uncompressed

[worthy cape]

They are the range when decompressed.

First block is for 0-256k decompressed, second for 256-512k, etc

[simple ravine]

how would I then know which blocks that are covered b the file-data if I don't know where that'd be?

[rapid pagoda]

1. the file descriptor in the bundle index tells you the offset and size in the uncompressed bundle data, which you can map to a range of blocks using the block granularity in the bundle header (usually 256k)

2. the bundle header contains a list of compressed block sizes; take a cumulative sum of those values and you'll get offsets into the compressed bundle data

[worthy cape]

The offset size tell you which part of the decompressed data a file is in. Each block would result in 256k of decompressed data if decompressed. You can randomly decompress blocks.

[simple ravine]

I am clearly having one of those days..

[worthy cape]

If your file starts at like 512k offset and is 384k large, you would start with the third block and decompress two blocks to get enough data.

I would draw you an image but I’m on mobile 😄

[simple ravine]

but that's 512k when uncompressed

will you be back at the pc later today? in case I'm continuing to have one of those dropped-behind-the-barn days?

[rapid pagoda]

if it helps any, the process for creating a bundle is effectively:

1. take a bunch of files and smash them together, recording where each one ended up
2. break that blob into 256k chunks
3. compress each chunk individually
4. smash the compressed chunks together and record how large each one was
5. that's a bundle

[worthy cape]

Shortly.

[simple ravine]

no rush, appreciate the help when time is given

what confuses me though, is that uncompressed offset/size is given in the file_info records

[golden bane]

You guys make me want to work on my own GGPK reader again 😂

[rapid pagoda]

right, because that determines which of the chunks the file ended up in

[simple ravine]

I tried to decompress the whole file and slice it using offset and size, but I got what looks like an incomplete file, hmm

[golden bane]

Fun bug I had: since I don't know anything about c/c++, I call ooz from the command line instead of turning it into a library. When reading the index bundle, I got only nonsense. Turns out, I forgot I had to wait on ooz. I just let it spawn a new thread and immediately proceeded to read from the file 🙈

[rapid pagoda]

You might need to pass each compressed chunk to the decompressor individually. I'm not sure Oodle supports concatenating compressed data

[worthy cape]

Each compressed block corresponds to a 256 KiB range of the decompressed data. You can decompress any block to fill in its corresponding section of data. You decompress the blocks you know you need to extract the data ranges that map to the files.

For file A, blocks 0 and 1, for file B, block 1, etc.

[worthy cape]

Concatenation is fine, you can bulk decompress a whole bundle I believe.

There's interior headers it understands.

[simple ravine]

yeah, I preallocate enough data for the whole file, and run ooz per chunk

[simple ravine]

ah, now it clicks, every uncompressed chunk is 256 bytes, except potentially the last

[worthy cape]

Hrm. I decompress block-wise. I wonder if that's required.

[simple ravine]

duh

[worthy cape]

256 KiB, but yes 🙂

[simple ravine]

typo yes 🙂

[worthy cape]

The upside of writing this code once last year means I've forgotten most about it 😄

[simple ravine]

haha

yes, I've looked at the project, and closed it several times

but once I am done, I hope I feel a bit accomplished by that, and hopefully it'll be of use for someone

inb4 poe 2.0 will have a completely different way of storing things

@rapid pagoda do you perhaps have some .txt-file or something extracted from a bundle that I could use to compare the results?

I tried installing the pogo thing, but i apparently need gcc as well

[worthy cape]

If you want something compiled, https://zao.se/~zao/poe/bun-x64-Release-g462332b.zip

[rapid pagoda]

wait what the hell that didn't extract properly??

[worthy cape]

bun_extract_file extract-files Content.ggpk C:\Temp\ Metadata/minimap_colours.txt

[simple ravine]

Ah, great thanks @worthy cape

[rapid pagoda]

well apparently I need to investigate what's going on with minimap_colours.txt but for now here's Metadata/Basin.ao

which, fwiw, is in the bundle _Tiny at offset 0x39c75f

[worthy cape]

My minimap_colours.txt seems to be 251 bytes, SHA256 a16669cd7be71fa145389106d3fd7f19c31ecb9eefd40d90dd074208a19dbab2

[simple ravine]

thanks!

et voila!

getting closer

[rapid pagoda]

huh, that is the same content I was getting for minimap_colours, maybe what I was getting was correct

[simple ravine]

yeah that is what zao's application extracted

however, in Basin.ao, I am getting some kind of potential BOM?

??version 2
extends "Metadata/Parent"

AnimationController
{
        metadata = "Art/Models/Terrain/Doodads/TempleDungeon/basin.amd"
}

[worthy cape]

Very common.

[rapid pagoda]

that's correct, a ton of files in the app are utf16

[simple ravine]

ah ok

[rapid pagoda]

enough that there's a pogo cat --utf16 option to autoconvert utf16->utf8 🙂

[worthy cape]

I need to figure out a web API for searching historical filesets some day.

I wish I could have something largely static, but for queries like "get me all of file Blargh.dat from first to last release", I guess I'm going to have to have some code running.

I want to get this file access API up and running so that I can start messing around with DAT specs again.

[simple ravine]

I remember I met Blocket.se-developers at some code-meet in Stockholm once. They said (at that time, it's a long time ago) they wrote C-modules for Apache that powered their site

or if it was nginx, don't recall

[worthy cape]

My intent right now is to put the data storage on the local web server behind Cloudflare, and expose some janky little API for searching filesets in tree and time.

[simple ravine]

is there some kind of odata-app that you could just glue between the webserver and database perhaps?

[worthy cape]

Got no idea how to express this.

Maybe just some "here's bespoke JSON with a "query", plz respond"

Or just expose postgresql to the internet.

[simple ravine]

bobby tables will like that a lot

[worthy cape]

"Worked" for Super Meat Boy and their mysql leaderboards.

[simple ravine]

well, it's kind of what odata is for tho, no?

it's just expressed in query string rather than as json

either that, or perhaps some graphql thingie between nginx and sql

[worthy cape]

Anyone saying Mongo gets a whippin'

[golden bane]

Just some endpoint you can post JSON to would be the easiest to implement I think. Maybe with an OpenAPI spec if you're feeling generous 😄

[worthy cape]

Out of curiosity, how many of you folks are IPv6-enabled?

[golden bane]

I have a static IPv4 address

[frank drift]

full ipv6 enabled afaik

[rapid pagoda]

I am. But, FWIW, Cloudflare will happily proxy v4 to a v6-only origin

[worthy cape]

Ooh, that's neat.

Very elegant links: http://[2a00:66c0:8:0:386b:6aff:febe:d840]/data/a16/a16669cd7be71fa145389106d3fd7f19c31ecb9eefd40d90dd074208a19dbab2

I wonder if it would be costly to have a rewrite rule that slices out the first three nibbles to make the URL more independent of actual storage layout.

[rapid pagoda]

Nah. Simple rewrites like that are cheap

[rapid pagoda]

fwiw I'm looking into making github generate binary releases for pogo 🙂

[worthy cape]

Oh right, I need a TLS cert for this junk.

[worthy cape]

Windows, the platform the Go people would rather not existed 😄

[simple ravine]

hehe

[rapid pagoda]

current status

[grave wren]

😄

[simple ravine]

classic haha

[candid ridge]

Is there a replacement for poe trade enhancer? Something that normalizes gem prices would be nice it's what I mainly used it for.

For gems show the cost of reaching max quality by buying the gem and the necessary gemcutter's prisms, also enabling sorting by this value.

or if somebody knows how to update it so it works with the new skill gems

[grave wren]

@earnest radish you sure kintsugi is gambleable? isnt it proph only?

[frank kayak]

web api? so ...go?!

i mean you should use a language where its suitable

[worthy cape]

It's more a matter of the interface, implementation language would be whatever I know and want to use.

Go... not high on that list 😄

[frank kayak]

go suchs atm for gui, ml and embedded and a few others

go is perfect for apis

rust async thing for sushi?

i mean rust shines in performance relevant stuff - stuff i wont touch

[worthy cape]

Perl CGI it is 😉

[frank kayak]

why not try sed? pretty light weight

[worthy cape]

I know a dude that would definitely try to use Awk.

[frank kayak]

just use some named fifos to pipe stuff in 😄

awk for structured stuff, sed for mess

and if you want to cause a mess

my first scripts were mostly sed - parsed train times - that was nice

at some point i noticed that xpath exists and is probably a better fit...

that led me into trying to implement an xpath parser in c++

at first i liked it

until i didnt

but mostly because of build things and not the language

hand written hacky Makefiles, making sure autoconf for the libs you need dont fuck up. lots of different copyleft license that wont work together and such stuff put me off a bit :/

[worthy cape]

There we go, origin CA set up, probably should get my host a fixed v6 address as this temporary one is, well, temporary 😄

[velvet fog]

trying to add basic markdown editor https://poedb.tw/us/Instilling_Orb

[worthy cape]

CF proxy works like a charm, after realizing that I've got it set up to be mega-strict about TLS. \o/

[frank drift]

@earnest radish I've found a few different items that match '75' so you're definitely right to avoid numbers

[rapid pagoda]

Not done yet, but done enough to show off: https://oriath.net/audio-new/Gwennen_the_Gambler/

Another part worth noticing: https://oriath.net/audio-new/Conversations_HeistVinderiNilesSelection2/

[simple ravine]

using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;

var app = WebApplication.Create(args);

app.MapGet("/", async http =>
{
    await http.Response.WriteAsync("Hello World");
});

await app.RunAsync();

All you need to use c# / .net to get started

that's the entirety of Program.cs nothing else needed

oh, and VS 2022 has Hot Reload, so you don't need to restart your app when you're writing code

[vapid pulsar]

This is a good book for how to approach design as a dev. I can send my digital copy if your interested.
https://www.goodreads.com/en/book/show/43190966-refactoring-ui

[grave wren]

its always the indexing :<

3d firewall 🚤

[oak estuary]

EmmyLua dies in the new update, yeah...

Mine got disabled, surprised yours still runs

[simple ravine]

finally

[fervent carbon]

Anyone know of a tool that manipulates your filter, hiding or showing certain currency items if you're low on them?

like the chaos recipe tool, it reads your stash tabs and hides / shows items, but for currency

[frank drift]

I have a tool like that

I don't use it as much anymore now that filterblade syncing is relatively painless

the cache delay this patch sucks though

[fervent carbon]

the more I can offload onto automation the better

[gilded tree]

I couldn’t find it in older messages, what do you want to do? A parser? Or do you have something different as a goal?

[simple ravine]

A ggpk/bundle/dat etc parser/exporter

finally got around to add bundle support

[simple ravine]

https://github.com/andreandersen/PoeSharp

[gilded tree]

Ahh

[simple ravine]

It's been on-and-off for a few years now

still WIP to some degree, but it's stabilizing a little

[whole ruin]

Anyone got a quick example of using fetch or any other javascript to get stash tabs from an account (with poesessid)? My attempt is getting 403d 24-7

[worthy cape]

If you're doing stuff from a browser, you may be smacking into some cross-origin shenanigans.

[whole ruin]

I am indeed

But i seemed to solve cors errors earlier

by using no-cors in header

now im just getting 403 forbidden

[worthy cape]

Maybe the site doesn't like you and your referer?

(I'm rather clueless in this area)

[whole ruin]

Im assuming its me not creating headers correctly (regarding cookies) as I would think since I can manually open the API url in a browser and get results it should be fine if i can recreate that in a javascript call

[worthy cape]

Browser's Inspector should tell you all about your request, shouldn't it?

[whole ruin]

yes they are not completely equal

hence why im asking if someone has an example they know works

[rapid pagoda]

That won't work. https://stackoverflow.com/a/43268098

The TL;DR is "you can't". If a web site hasn't explicitly permitted you to make a cross-origin request to it using Access-Control-Allow-Origin, you can't make cross-origin requests to that site and see the results.

The CORS proxy workaround mentioned in that answer won't help you, because that won't pass through authentication.

[whole ruin]

So the only way for me to do this is to either mail GGG for auth, or to write an extension that injects into pathofexile.com itself and call it i guess. I find it weird I can directly access the API call in browser and get json request but i cant simulate that in code

[worthy cape]

Pages are subject to a whole bunch of browser-imposed rules about what they're allowed to do. User-initiated initial requests, less so.

Standalone browser-like programs like Electrons and friends, also less so.

[rapid pagoda]

not finished, but finished enough to replace the old version -- https://oriath.net/Audio/

The index is a little incomplete, I don't have the Betrayal triggers showing yet, and I still need to re-import a bunch of transcriptions from the old site -- but it's already way better than what I had before

[whole ruin]

Guess ill mail them then. I just assumed apps like exilence etc worked just by using poesessid

[rapid pagoda]

Exilence works because it isn't in a browser and isn't bound by same-origin constraints

[whole ruin]

Does that mean a nodejs server/app would work as well?

[rapid pagoda]

If you wanted to have users run it on their own computer, sure.

Anything that involves passing POESESSID to any server other than pathofexile.com is an absolute no-go, though

[whole ruin]

nah i mean i was just prototyping this in a jscript file loaded in a html loaded in a browser. In the end i would use nodejs to get info from POE API then query that nodejs own API.

or i can just write an app to test this for myself first i guess

Ty

ill try it out that way

[whole ruin]

ye got that working straight away

[whole ruin]

Again ty for help, I can finally list bulk expedition currency without having to spend time calculating all that shit out

[timber path]

@whole ruin poe overlay community fork uses the poe sess Id to retrieve stash info, it uses electron (node js + chromium) & angular with typescript code. So feel free to look around the code, or ask me what you want to know.

[velvet fog]

https://poedb.tw/us/Expedition_league
markdown edit, You need to log in to patreon(No subscription required) and send /u/chuanhsing to get editing permissions.

[frank drift]

basically a wiki?

[velvet fog]

I know that some websites use iframe embedded wiki sites, but the installation and management of wiki is not suitable for me, so I finally decided to use simple markdown.

I will provide some basic tag for item/unique/gem, but not full wiki functions

[simple ravine]

@civic crane the until property in the dat-spec, what type is that? all of them are currently null

[civic crane]

as @worthy cape suggested it can be used for versioning, but not in use for now 🙂

[simple ravine]

ok, so probably a string, or a float?

like "3.15" or 3.15

[civic crane]

string

[simple ravine]

ok!

the column names being null instead of "unknown0" etc is throwing me a wrench 😄

[timber path]

Is anyone actively maintaining some dat spec somewhere? Maybe the PyPoe fork dat spec is still being maintained?

[woeful sphinx]

would be great if https://github.com/poe-tool-dev/dat-schema catches on for that, but currently it is behind others

I just started an importer from that repo to the PyPoE format, so the PyPoE spec won't need to be maintained independently anymore soon (hopefully)

[civic crane]

but currently it is behind others
behind what? so much pypoe forks, I can't track all of them

[woeful sphinx]

to fix the spec errors I just got, it wasn't sufficient. I looked at https://github.com/aianlinb/LibGGPK2/blob/master/LibDat2/DatDefinitions_extra.json and some changes posted in #wiki (see https://github.com/brather1ng/PyPoE/issues/6), no idea from what spec those came from.

but once I get the importer done, I'll open PRs in dat-schema for the differences

[civic crane]

yeah, it's more like I first fix errors for what is needed for Awakened and in free time left lurk in other tables. If everyone can update schema for tables they are interested in or at least say which tables they need to be fixed asap we can maintain it

[long sky]

FWIW I've got a wiki fork up and running now

https://www.poewiki.net/wiki/Path_of_Exile_Wiki

[velvet fog]

got it

[timber path]

Yeah, that would be great. I mainly asked about dat maintaining because I've just picked up some slack on my own asset updater for poe overlay CF, and now got to the point where all what's left is "fix" or rather complete the dat defs to avoid "remaining bytes" warnings.

To do so, I've expanded on my poe asset visualizer tool to show me "remaining bytes" in different formats to try and decode what they mean

Example of how I display "remaining bytes"

With this addition, I've been going through a number of dat file specs I've been using to generate poe overlay CF assets, and "discovered" some fields that are currently not present/added in @woeful sphinx 's PyPoE fork; and likely also missing from others

[woeful sphinx]

yeah, my fork is probably missing a bunch of fields in the tables not used by RePoE. that's the issue with most specs, I guess: only the things the corresponding tools needs are updated.

that's why I'd be glad if everyone would pool their dat spec reverse engineering into one repo (https://github.com/poe-tool-dev/dat-schema)

[vestal pond]

That file is not fully updated, you can take a look at this

[woeful sphinx]

thanks. I saw that there is some merging going on, but it had what I needed.

[simple ravine]

Ok, so now that several are here

can we find a common way for this, I've been proposing this for years 😄

[timber path]

I'd like to help and contribute, but my "decoding" skill aren't that good. Most of the time I can't even tell if I encounter a ulong or list of ulongs that it would reference a base item type, or mods, or some other table... so if others are willing to share their techniques and insights on how they go about it, then I can learn from that and be more helpful

[simple ravine]

I think the first thing would be to agree to 1 place

and everyone contributes to that place

[woeful sphinx]

honestly, what I've been doing just boils down to adding ints/ulongs/bools until the spec size is correct. so I wouldn't even call that decoding 😄

[woeful sphinx]

I propose the place @civic crane created for exactly that cause: https://github.com/poe-tool-dev/dat-schema

[simple ravine]

I started implementing that format as the dat-spec for poesharp yesterday. I'm not familiar with the graph ql thing, but it can't be that difficult

[timber path]

yeah, that's where I'm at right now, and what I'm doing now. Mostly marking things as "unknownX"; but a next step is to "link" an unknownX to e.g. base item type, or mods, etc... or even just properly name it; that's the part I'm lacking knowledge mostly though

[velvet fog]

The fifth new PoE Wiki: PoEDB Wiki mode

[worthy cape]

What's the aim for that, is it to have the introductory/educational text up-top on each page, or something more granular?

[grave wren]

sixth wiki will be asciidoc static site gen

just wait

[velvet fog]

just add some introduction and join the war

[grave wren]

people will love it

[carmine merlin]

PoB wiki next /s

[worthy cape]

I'm not making a Markdown renderer 😄

Unicode support in plain text is hard enough ^_^

[grave wren]

no you'll do the asciidoc renderer

[velvet fog]

https://www.reddit.com/r/pathofexile/comments/p0gt70/the_fifth_new_poe_wiki_poedb_wiki_mode/

[grave wren]

poedb supporter packs when

cant just go with indie website when you dont offer supporter packs!

[oak estuary]

I love how the thumbnail is Fandom for that post

One thing I like about the current wiki (and proper wikis in general) is having a running history of changes so I can go back and see when an item or skill was buffed/nerfed

[worthy cape]

One of the things I feel about these header kind of things is that I have to scroll an indeterminate amount down to the information I "really want", the tabs and data.
Can there be a button/way to "jump down"?

Like how say Imgur has a button to jump down to comments below an album instead of scrolling your butt off.

[velvet fog]

I will add this

[civic crane]

I think this is more like notes than a full wiki on poedb, and as a Notes should be down on page.

[swift beacon]

I'm all for using https://github.com/poe-tool-dev/dat-schema for the DAT spec, but there's a couple of fields missing that my tool uses that were in PyPoE, such as file_path and file_ext.

Maybe I'll have to finally figure out how to create a GitHub account so I can contribute.

[civic crane]

If your tool needs additional ...
#tooldev-general message

[timber path]

If you want to dat-specs fun, feel free to join my quest to figure out the new specs for "MonsterVarieties" 🙃

[swift beacon]

Yeah, I need that one working for my 3D model viewer.

[civic crane]

poe-tool-dev/dat-schema has them up-to-date, or you mean find names of new columns?

[timber path]

It doesn't seem to be up to date, unless I compared them incorrectly to the specs I got locally 🤔

[timber path]

Figured it out Special_ModsKeys was still in my (and PyPoE) specs, but it got removed

[swift beacon]

So how does the versioning work? It looks like the files with a version number in the name are the tables added in that release, and _Core is the rest of the tables?

[civic crane]

yep, @until / @since directives should allow more detailed versioning. But I don't see point of implementing them right now until zao allows us to use his versioned storage 😄

[worthy cape]

Simply know the hashes and access elegant URLs like https://poe-data.zao.se/data/a16/a16669cd7be71fa145389106d3fd7f19c31ecb9eefd40d90dd074208a19dbab2 😉

Saying so, nginx and/or CF seems to have crapped the bed ^_^

[civic crane]

stealing ideas from novyn I see ||gen/image||

[worthy cape]

Very odd. It responds to the internal v4 and public v6 address, but CF can't talk to it.

Ah, my v6 connectivity seems hosed.

In any way, it's all hosted internally and as you can see, doesn't work 😄

A-ha. Didn't have RA for the default route.

Worst thing about all this is that I have no reliable way to automatically fetch new data.

SteamDB by design doesn't have any APIs, and Steam loves to send confirmation tokens to your email when trying to login.

[oak estuary]

Since we were on the topic, does anyone have a more complete spec for BuffDefinitions.dat? Or an idea of how they link to skills? @velvet fog I see you have the buff linked to the skill on PoEDB (e.g. https://poedb.tw/us/Petrified_Blood#petrified_blood ), but I can't find the field that links them...

[swift beacon]

So I would add file_path and file_ext like this:

type ActiveSkills {
  Id: string @unique
  DisplayedName: string
  Description: string
  _: string
  Icon_DDSFile: string @file_path @file_ext(file_ext: '.dds')
  ...
}

[civic crane]

not against of adding file_path, but why you need extension name at schema level and not when you actually read contents of column?

[swift beacon]

Man, discord formatting is messing me up 😆
Edit: think I figured it out, it was the _ in the first line messing everything up.

[civic crane]

use tripple backticks (on new line)

[swift beacon]

The reason the extension is handy is because some columns that are file paths omit the extension for some reason.

But I guess there's other ways to figure out what it should be.

I think some might omit the extension because there's multiple files with the same path but different extensions.

So it acts as a link to multiple files

[simple ravine]

Plenty of those already out there 🙂

[worthy cape]

Fewer suited for OpenGL in a C++ program, or worse, Lua.

[simple ravine]

I'm confident there are HTML renderers for OpenGL in a C++ program tho

so u can do MD->HTML->render

[oak estuary]

If only we were displaying to HTML, lol

[simple ravine]

Another level of indirection / abstraction is the solution to all programming problems

[worthy cape]

Still not sure how to make the filesets searchable.

[simple ravine]

What do you want searchable?

[worthy cape]

Got enough DB for this at least:

[worthy cape]

I don't know that either.

Like "all historical versions of file X". "all files in subdir Y for manifest Z"

[simple ravine]

What I tell architects at work:

My golden 2 rules for successful architecture is

1. Defer decisions as much you can into the future.
   Rationale: The more information you can gather before making an architectural decision, the better decision yo u can make.

2. Simplicity is your friend.
   Rationale: Obvious

[worthy cape]

Manifests have a natural order to them so I could do ranges between manifests, but preloads don't.

[simple ravine]

Gather your information 🙂

brb, my son wants to try his new soccer ball

[simple ravine]

Add a synthetic datetime column inferred from when the patch was released

[worthy cape]

There's timestamps for the changeset which also relates app buildids and the associated manifest ids.

Might put in tables for all of those, assuming I can get them out from steamdb or steam easily.

Worst thing about all the releases is that there's no good source of PoE version numbers for anything, the wiki is the best, with which you can correlate timestamps.

[simple ravine]

It was the wiki I was thinking of

not sure how perf is, but 3NF and computing ad-hoc in query like the encode(decode(... kills perf really quickly

[worthy cape]

That was mostly because I was silly in the past and stored hashes as base64 instead of binary or hex-text.

[simple ravine]

I'm surprised that "big data" uses snowflake-designs similar to 3NF

[worthy cape]

The dedup of paths and manifests to IDs I still hold is a good idea.

[simple ravine]

It's definitely a cool idea 🙂

[worthy cape]

Looking forward to building all sorts of query DoS.

"hello server, plz2give all file info for all time"

[rapid pagoda]

personally the ones I'd care most about would be:

1. "In what versions was a file with this name added/removed/changed?"
2. "What paths has this content hash existed at, and in what versions?"
3. "What file names have ever existed in under this directory prefix?"

[worthy cape]

2. there is a fun one, the web of equal-content can spread out quite a bit for small and common contents

[oak estuary]

I just checked that spec, claims to have a column between OffHand_ItemClassesKey and Helmet_ItemVisualIdentityKey, but it looks wrong:

With column:

[civic crane]

my bad, indeed

[rapid pagoda]

@swift beacon btw, as our resident Loremaster, you may be interested in my recent update to https://oriath.net/Audio/ -- it's now updated with all Expedition audio, and now organises lines by conversation triggers

[woeful sphinx]

Helmet_ItemVisualIdentityKey is still there the unknown field before it is gone

[swift beacon]

Thanks, it's helpful to have them organized like that.

[oak estuary]

Do you happen to know where this key went or if it was rolled into another column?

[velvet fog]

@oak estuary they have same name in this page

[worthy cape]

The curse of @simple ravine is upon me. SteamKit is .NET 😦

[grave wren]

Praise the omnissiah for the .net lib

[simple ravine]

thewaht

[grave wren]

The Machine God, also known as the Omnissiah or Deus Mechanicus, is the supernatural entity worshipped by the tech-priests and other servants of the Adeptus Mechanicus as the embodiment and bestower of all knowledge and technology in the universe.

WH40k is fun

[simple ravine]

oh

[timber path]

No clue at all. It was in my specs, but not in the "central specs", so I removed it from mine and everything seemed to work and look just fine again

[fickle yew]

https://github.com/SteamRE/SteamKit/wiki#ports

[coarse token]

@carmine merlin Sorry to ping you/bother you, was just curious about the new PoB update. I didn't notice anything about updating the damage values for Absolution minions in accordance with the hotfix they received earlier in the league. Is this something in the works or is there a way to simulate it within PoB? Or did it possibly get thrown in without a note. Apologies again for the ping

[worthy cape]

C++ one seems like a nice 8 years old, and I'm a bit afraid to use others as they tend to be behind on functionality.

Considering improving my copy-paste workflow from SteamDB to something more automated.

[carmine merlin]

It's hidden server side or in some file that I don't know the location of so that's why it wasn't in the update

I could add a psuedo more multiplier to the skill but it wouldn't be 100% accurate

[gilded tree]

I tried the rust one some time ago, sadly some features were missing. No clue what the current state is

[simple ravine]

It's obvious c# reigns supreme 😄

[worthy cape]

I don't suppose you have any favourite Postgres bindings for .NET?

[simple ravine]

I'd probably use the one Marten is using (martendb.io)

or the one Entity Framework Core uses, let me check, but I think Microsoft made one as well, because they provide EF Core support for pgsql themselves.

(and offer it as a service in Azure)

https://github.com/npgsql

If you want Repository Pattern / Unit of Work, use EF Core

if you want to hand-write your stuff, you can use Dapper

[simple ravine]

The difference between dat and dat64 are the width of reference pointers for local rows, anything else?

[rapid pagoda]

Width of pointers into dynamic data, including strings, and width of row IDs

short (32-bit) row IDs become 64 bits, long (64-bit) row IDs become 128 bits

[simple ravine]

when you say width of Row IDs, you mean what?

[rapid pagoda]

The integers which reference a row in another data file by its index, and whose value is 0xFEFE…FE to represent a null value

[simple ravine]

ah

public bool Is64Bit => DatType is DatType.Dat64 or DatType.DatL64;

time to add 64 bit support

while I'm at changing this to support the new spec format

much needed overhaul.. 3rd one at that, at least.

public static int GetSize64(this ColumnType type) =>
    type switch
    {
        ColumnType.ForeignRow => 128 / 8,
        ColumnType.Row => 64 / 8,
        ColumnType.Array => 64 / 8,
        ColumnType.Bool => 8 / 8,
        ColumnType.F32 => 32 / 8,
        ColumnType.I16 => 16 / 8,
        ColumnType.I32 => 32 / 8,
        ColumnType.I64 => 64 / 8,
        ColumnType.I8 => 8 / 8,
        ColumnType.String => 64 / 8,
        ColumnType.U32 => 32 / 8,
        ColumnType.U64 => 64 / 8,
        ColumnType.Unknown => 0,
        _ => throw new NotImplementedException()
    };

does this look correct, or do I need more information to calculate the size?

[rapid pagoda]

Array should be 16 (128/8). 64 bit count, 64 bit offset

Yes, it's ridiculous.

[simple ravine]

https://giphy.com/gifs/just-blinking-3ELtfmA4Apkju

I saw that there are some references that references ID columns in other tables, are those also 128 bit for 64 bit dat files?

like do have to discriminate between a key reference and a row reference in a foreign table?

forgot how it was all done, 3 years ago now

[rapid pagoda]

https://github.com/oriath-net/pogo/blob/e8243e03/dat/types.go#L56-L98 is my equivalent, fwiw 🙂

[simple ravine]

ah great

ColumnType.Row => 64 / 8, is correct though, for 64bit yes?

[rapid pagoda]

Yeah. That's what I refer to as ShortID

[simple ravine]

ah ok

[swift beacon]

Thanks to everyone's help updating the DAT specs, I got my model viewer running again. The bad news is, there's a new SMD version this league, so I can't read the 3.15 NPCs and monsters yet.

[simple ravine]

oh cool, and ... damn 😐

I think @worthy cape was doing something similar?

[frank drift]

t-posing just like in game

[swift beacon]

😆 Yeah that was the best bug.

[simple ravine]

is that the same for all the 128 bit ones? a bunch of zeroes before a 64 bit value?

[rapid pagoda]

Yeah. At least it compresses well…

[simple ravine]

haha

game developers and their data-driven development and memory alignments, I suppose

or well, the 128 bit thing doesn't make sense anyway

[civic crane]

Its not alignment, it's reserved space

[rapid pagoda]

Could be space for a fat pointer of some kind

[swift beacon]

At least the error messages look neato.

[worthy cape]

New format, eh?

Been ages since I touched the models much, most the recent graphics work was on hideout-related ones and since then I've been down the dedup/dat hole.

[swift beacon]

Yup. Seems like they started using it for some models in 3.14. Haven't tried figuring it out yet, so not sure if it's going to be simple or not.

At least I can now pan and zoom the error messages in frustration. 😆

[worthy cape]

So looking forward to when I can do some format parse sweeps across game releases again.

Have you've made any progress on getting materials and their generated shaders mangled into something that your web graphics can understand, or is it mostly just rendering the textures as-is?

I never made too much progress on shader synthesis from the fragments.

[swift beacon]

I'm just rendering the textures as-is. Graphics are way outside my understanding, and I have no real desire to get into it, at least at this point.

[vapid pulsar]

Just took a quick looks at the new SMD format and it looks like a new section was added.
If you read an extra 4 byte int after the bounding box the buffers line up again. Looks like there a new structure in the tailing section now too that this int is most likely representing

[vapid pulsar]

Looks like it's some structure for collisions

[swift beacon]

Yup. Literally just had to read one extra int.

[rapid pagoda]

where do I get that badass skull buckle belt

[swift beacon]

You could 3D print the model.

[worthy cape]

So many tools to write, so many formats to sort out, so little time.

I'm like five yak layers down atm.

[simple ravine]

you just have to get a passive income, quit your job etc so you can dedicate your life to poe open source tools 😄

[worthy cape]

Got almost nada done for the summer vacation.

[simple ravine]

July had good weather though, at least here, so that distracted me

[worthy cape]

Plan right now is to make something bare-minimum that pulls changelists from Steam, jam it in a DB, and auto-download manifests. Once I've got that rolling I can start focusing on ingests and exposing filesets.

[simple ravine]

My end goal is yet-another-game-companion, because I have a bit of hubris thinking I can make the best one. I've been stuck developing a library parsing things for 3+ years though, so yeah heh...

But hey, WinUI 3 is Soon ™️ out

[worthy cape]

I have no idea how to package up something like this and make it deployable, fearing that the answer is either container swarms or Ansible.

[simple ravine]

Isn't this slightly comparable to what you do at work?

If I would do it, I would have used a serverless function (Azure Functions in my case) that would poll every N period of time, and kick off a batch job (Azure Container Instance) to download the data and ingesting the data into a database of some kind.

[worthy cape]

Primary work is building software in a big module tree.
Cloud stuff is mostly poking at an opaque OpenStack that sometimes works.
Regular sysadm junk with Puppet and stuff, I'm super out of touch with.

[simple ravine]

But if you want to do it on-premise, I'd potentially try K8s with knative / keda

or just a couple of containers on some random VM to start with

[worthy cape]

Right now it's running on a single proxmox machine, with database in a LXC container and downloader/ingester in a VM.

All data storage on proxmox server and a subset on the NAS.

Very hand-setup tho.

[simple ravine]

I have no idea with a proxmox is, it sounds like a character from an animated tv show 😂

[worthy cape]

VM cluster, think esxi but less meh.

[simple ravine]

ah

just get some vSphere and call it a day 😄

I can sell you some OpenShift licenses!

[rapid pagoda]

the buzzword-compliant solution these days would probably involve kubernetes

[worthy cape]

The icky thing is the bulk data storage needed, services themselves can run reasonably easily.

[simple ravine]

Store the data in a blob store somewhere and just store the meta-data in the database then?

but I guess that can become somewhat costly as well, if it escalates

[rapid pagoda]

https://min.io/ might be worth looking at

[simple ravine]

You can get 7 TB of archive-level blob store for €10 in Azure

[worthy cape]

I guess the big 12T storage is mostly archival, actual live ingestion is way more on the fly; just two separate stores of dedup:ed bundles and dedup:d exploded files.

Could ignore everything that deals with legacy GGPKs, just prebake that once and for all.

Now smeep, work tomorrow 😄

[simple ravine]

gngn!

[empty venture]

Hello guys, got a quick question. Do you know if it is possible to get items' 2D assets anywhere, preferably using base item name and/or unique name? I saw that poe.ninja is getting their icons from web.poecdn.com. If I understand correctly, those come from the official API when requesting ladder/trade data. Can you get icons for items that don't exist in the game - for something created in PoB?

[timber path]

afaik not in an easy way when thinking about long-term support of such a feature. You can currently compose the URL based on the internal data structure, but that feature is being removed next league by GGG. So you'd have to go through public stashes, hope said item is in there and grab the image URL generated by GGG. If it's not in there, then there's currently no way to figure out the URL yourself (since it has a random hash in it, likely to avoid creation of custom urls based on the internal data)

[velvet fog]

for base weapon and armour, you could try https://www.pathofexile.com/item-data

[empty venture]

There isn't any external API that would make those available? That sucks

All the icons are inside of ggpk, right? So technically one could grab all of them from there?

[carmine merlin]

@velvet fog have any idea where GGG added the buff for Absolution in the GGPK? Minion and its skills looks exactly the same for me

[velvet fog]

no idea, I don't keep previous version https://poedb.tw/us/Absolution#SentinelofAbsolution

[candid ridge]

if you're looking to replace the poewiki that's something you might want to consider supporting in the future

[velvet fog]

i will keep in mind

[hazy fiber]

I mean realistically only things mentioned in patch notes would need to have that, and only the patchnote itself would have to be stored

and maybe legacy items

as thats all the current wiki has

[simple ravine]

What would be really nice would be an open source markdown-esque wiki that everyone can contribute to, and is not affiliated with any company that owns that information and infrastructure

Potentially with some knowledge graph data of some kind. Licensed under MIT or similar, so other tool developers could leverage and contribute to it.

[long sky]

MediaWiki does have a queryable API that might be useful?

[simple ravine]

That API is meh, and the content is proprietary (even though it's community contributed, you give it to the company who owns the wiki) with a restrictive license IIRC

[grave wren]

isnt mediawiki foss or at least open source and can be selfhosted?

[long sky]

Yeah, super easy to self host

the content license depends on the wiki, it can either be fully copyright, fully open source or anything in between

[simple ravine]

@civic crane when type is array in the definition, what is the inferred data type of that array?

[velvet fog]

https://poedb.tw/us/Syndicate_Operative

[worthy cape]

Array-of-unknown? https://github.com/poe-tool-dev/dat-schema/blob/89051f17c11e06528c3a4e18cceba732ce1080cb/src/types.ts#L20-L21

[simple ravine]

yeah

i see it in a couple of places, not sure how to handle it

[hazy fiber]

would it not be better to make the graph logarithmic?

as in the y axis be somthing like
1k, 5k, 25k, 125k, 625k, 3mil, 15mil

[civic crane]

Like array of bytes

Thanks to eingin we know them now

[woeful sphinx]

@civic crane I see that references to enums in the schema are specified as i32 (or u32 in some cases). is there a difference between that and referencing the enum? e.g. MinionActiveSkillTypes: [i32] vs MinionActiveSkillTypes: [ActiveSkillType]

[rapid pagoda]

fwiw: as a stylistic thing I try to use i32 for columns where the sign bit is unused; only use unsigned types for columns with weird not-quite-numeric values like hashes

[simple ravine]

Live Unit Testing ❤️

[civic crane]

Because in pypoe enum defined as python code, they were exported as i32, but should be converted back to using enum syntax (I think I defined only 3 so far).

Also from what we saw on stream they are not using unsigend ints at all. And as chuan said should remove 8, 16 bit sizes too

[rapid pagoda]

There's currently four references to u16/i16 in current formats, but I bet most of them are pairs of unused booleans. NPCMaster, Pet, SynthesisBonuses, SpawnObject

Tons of u8s, with a lot of them explicitly named "FlagX"

[velvet fog]

I have no u8 and u16 in spec file

Supported .dat data types are also shown: String, Decimal, Bool, File, Directory, Reference, Interval, Color, Point, Enumeration and DateTime

[simple ravine]

The UI don't seem to distinguish from int/long/floating points in that dropdown

[velvet fog]

long only for reference

[rapid pagoda]

What's the application name in the window titlebar? Can't see it behind the stream info 😆

(or: got a link to the vod?)

[simple ravine]

https://www.twitch.tv/videos/1107637286?filter=all&sort=time

ops too slow

[rapid pagoda]

first link was dead anyway -- preview must be old

and I don't see that content in the second link

[simple ravine]

why did they delete it 😦

[timber path]

WPF's DataGrid is dang slow as hell if there's a ton of rows; but it does look nice & more human-readable with reference fields resolved

[simple ravine]

@timber path do you use virtual mode on the datagrid?

[swift beacon]

Yeah that sucks, I really wanted to watch it.

[timber path]

Yup

[simple ravine]

.NET Framework or .NET 5/6?

[timber path]

It was a .NET Core 3.1 ; I updated to .NET 5 earlier today to "test" and see if it's better

[simple ravine]

interesting. should be fast.

[timber path]

Horizontal is fine. Vertical is the one stuttering like crazy

[simple ravine]

Hmm, are you doing any sync/lazy data loading? You're pre-populating the whole list right?

[timber path]

pre-populating a List<object>, then assigning it to ItemsSource

[simple ravine]

now I am curious.

[timber path]

(the objects are ExpandoObjects though)

[simple ravine]

that could be an issue

what if you transform them into something the gridview "prefers"? it's a long time since I worked with WPF, so the names are escaping me

[timber path]

Same here, it's been quite a while. But if you got some idea, I could try it. Or if you want to fiddle with it yourself, I can push my current changes

[simple ravine]

yeah, im curious

i have to run in a few minutes to get kid from soccer, and do some errands, but will be back around 18:00 my time (UTC+2)

[timber path]

Sure, no rush. I'll commit & push, and leave the link to the branch

[timber path]

Branch with the slow WPF DataGrid & ref fields (as shown in previous screenshot)
https://github.com/WhiteFang5/PoE-Asset-Updater/tree/feature/ref-dat-files

[velvet fog]

https://www.timeanddate.com/countdown/generic?iso=20210812T07&p0=2099&msg=Baeclast+x+GGG

[golden bane]

Never close your browser tabs 😂

[golden bane]

I still have an active session for the Jonathan talk VOD. Screen capping now

[worthy cape]

I didn't think about downloading it, figured they'd do their thing and upload it to YouTube eventually.

[golden bane]

Note to my future self: always download tool dev related stuff asap

[golden bane]

There are some videos on the Game Developers Guild's YouTube channel, but they're 4 months old so I don't think they're updated regularly

[worthy cape]

Twitch thing smells to me like they're not big enough to have arbitrary VoD retention.

[vapid pulsar]

Ive got the thing downloaded

I'll upload it now, but the thing is 2gb haha

[vapid pulsar]

https://www.youtube.com/watch?v=XNN0RLIfg-Y
Processing right now, should be up soon.

[vapid pulsar]

Damn

[worthy cape]

Do you have anywhere to put the raw video file?

[vapid pulsar]

I can upload it to mega. Unfortunately I just turned my PC off

And I need my sleep haha

[worthy cape]

No worries,

[broken cloud]

Huh, you have to verify your youtube account to upload videos longer than 15 minutes

[worthy cape]

@velvet fog I'm having a whole bunch of broken links from Google and page searches for pages like https://poedb.tw/us/mon.php?n=Jun%2C+Veiled+Master

[oak estuary]

I also have it downloaded, but I'm afk for a few days. I'm sure someone will have it up before I get back 🙂

[golden bane]

If anyone wants the VOD now, PM me

[dusk coral]

didnt really know the right channel for this, but has anyone tried installing poe on a ramdisk or enterprise optane? Is the load time client based or server based?

[worthy cape]

Top-tier labrunners routinely leverage ramdisks, not sure how much I/O it saves them.

Probably not too much of a difference from NVMe or having a hefty file cache.

Anything you read from disk is block compressed in bundle files, so you've got a few things that affect your load perf...

[dusk coral]

yea, I think windows 11 has issues behind the scenes with PCIe and cpu scheduler so I get weird lag when nvme drive first starts loading something, like a super quick HDD spin up

[worthy cape]

You've got the possibility of antivirus or other hooks messing with the opening of bundle files, you've got the fact that in order to randomly access a file in a bundle you need to decompress at least 256 KiB of data, even for tiny files.

The decompression also means that you're going to spend a fair bit of CPU on things, and you're going to pay page-wise both for the bundle reads and the resulting decompressed payload.

[dusk coral]

ive noticed the windows AV service chilling at 5% cpu, and I have a i9 10850k, the audio even does that staticy thing it used to do back in the day when cpu shits itself under load. Its exclusive to windows 11

[worthy cape]

You may also want to check with something like ISLC if you're starved for zeroed pages, if the OS has to scrub the standby list for them you might have some delays.

[dusk coral]

with my limited knowledge i have a conspiracy they are working out the bugs in a new cpu scheduler to get ready for intels big.little

[worthy cape]

But if you're hosed enough to have audio interrupts... maybe check with LatencyMon if something's krangling your system.

[simple ravine]

Did you have these issues on Windows 10?

[frank drift]

that was more lab racers than lab runners, I don't think runners care

[worthy cape]

Don't forget to have chipset drivers and stuff too.

[worthy cape]

Yeah, those.

[frank drift]

the dominant racers were winning through exploits rather than ramdisks anyway

[dusk coral]

nope, it goes away when I reboot, and seems to be exasterbated by certain programs also running like logitech crap and msi afterburner

[worthy cape]

I might be saved from having to do too much C# by the way.

[simple ravine]

but C# is love

[worthy cape]

Turns out that there's not enough historical information available from Steam, it like the GGG patcher mostly only cares about syncing up to current state.

Plan right now is to take a big dump of SteamDB's history page, mangle it through BeautifulSoup4 once to populate my database, and then run a changelist listener on the API to get new stuff.

[dusk coral]

I tried asking on reddit why big peripheral makers and pc mobo mfgrs cant figure out simple programs, they've been bad since I was old enough to use windows 95

[simple ravine]

truth.

[worthy cape]

@dusk coral Don't forget to take out all forms of RGB or other specialty software like ASUS Armoury Crate. They're extremely bad for system reliability.

[dusk coral]

logitech is the most guilty, they are a big company and the defacto corporate choice