@worthy cape https://bitbucket.org/zao/gggtools/ can't access
#tooldev-general
velvet fog
grizzled seal
https://poedb.tw/us/unique.php?n=Kingmaker
NearbyAlliesHaveFortifyTwoHandAxe9 and NearbyAlliesHaveCriticalStrikeMultiplierTwoHandAxe9 don't appear in Mods.dat for me, but NearbyAlliesHaveCullingStrikeUniqueTwoHandAxe9 and earbyAlliesHaveIncreasedItemRarityUniqueTwoHandAxe9 do
worthy cape
@velvet fog Made a lot of my legacy stuff private.
grizzled seal
as far as I can tell all of those mods are outdated and were replaced by new mods with different names when GGG nerfed the range from 50 to 30, all of the new mods are in Mods.dat
worthy cape
@velvet fog Made it public again.
velvet fog
thanks
https://pathofexile.gamepedia.com/Kingmaker
|explicit1 = LocalIncreasedPhysicalDamagePercentUniqueTwoHandAxe9
|explicit2 = LocalIncreasedAttackSpeedUniqueTwoHandAxe9
|explicit3 = LocalCriticalStrikeChanceUnique__9
|explicit4 = DisplayNearbyAlliesHaveIncreasedItemRarityUniqueTwoHandAxe9
|explicit5 = DisplayNearbyAlliesHaveCullingStrikeUniqueTwoHandAxe9
|explicit6 = MeleeAttacksUsableWithoutManaUnique__1
|explicit7 = DisplayNearbyAlliesHaveCriticalStrikeMultiplierTwoHandAxe9
|explicit8 = DisplayNearbyAlliesHaveFortifyTwoHandAxe9
grizzled seal
I'll update the wiki as these mods are clearly incorrect, 2 of them don't even exist and the 2 that do have the wrong radius based on in-game testing .. or maybe not. The display mods actually do seem to be the correct choice here, it's just the wrong radius is being inferred from them, it's definitely 30 not 50. Not really sure how to address that beyond just pointing it out.
worthy cape
Did some highly scientific diffs between the mod_types for the new patch - https://gist.github.com/zao/b993a9ac1f02456fcc48bf3a849d934b
Opened an issue on PyPoE for the changes in the Tags format, the new field is on the end and is a ref|string for you playing at home.
carmine merlin
@worthy cape Here's a list of the in game mod names with the tag changes if you're interested https://github.com/PathOfBuildingCommunity/PathOfBuilding/commit/4d088ef638f5147ec345b77eb56fd6a0a4a8c92d
worthy cape
Nifty!
Not digging too deep into it, the subject came up in a chat and I wanted to see if I could extract something ^^
rapid pike
Made this over the past few days
If anyone wants to check it out.. Still working on it and adding features
Exports to CSV format so you can use it in excel or docs
rapid pike
I got surgery in half an hour so I won't be able to comment on it but let me know if you have any advice or suggestions
golden bane
@earnest radish You don't need isinstance() for that, you should explicitly test for None like so:
if var is None:
# Your logic
All good ๐
zealous gazelle
Helo!
Who can I ask about the TOS and whether my program is allowed or not? ๐ I have been developing a horticrafting station extraction tool.
rotund solarBOT
Don't ask to ask! Post your question here so that people actually know if they can help. It's also better to discuss your question publicly so more people can chime in or learn! That's why these channels are here!
zealous gazelle
Alright then, So basically I have been developing this tool:
https://github.com/JeyR01/Harvester,
This let's people organize their crafts, and filter, etc,
(Yes I know there are already probably hundreds of other programs for this)
So basically I got a request from the people I shared this with, that It would be nice if they could just get their whole stash tabs full of horti stations in here instead of copy pasting all the time.
Now, I know of a way I can implement this, but this includes the use of pathofexile.com API, with POESESSID, username, and stash tab index. Already tested it and it works. I didn't publish it yet because I wanted to know 100% that this is not bannable.
So is this allowed or not ? POB also uses this API but I don't know if POB has any license for the use of it.
simple ravine
should be fine
zealous gazelle
sadly that answer is not enough ๐ , well at least not for me,
grave wren
then email ggg support
zealous gazelle
I thought that's why this discord is here
๐
it would be enough if someone who is already using it tells me it's fine, I'd be happy
simple ravine
"using it"
Using what?
zealous gazelle
the not-so-public API
simple ravine
I've been here for roughly 4 years. I think I have a decent understanding of what's kosher and not.
zealous gazelle
alright, ๐ I didn't know,
simple ravine
If that's not enough, e-mail GGG
zealous gazelle
in that case i'll accept your answer, thank you very much
grave wren
discord guys can just guess
ggg is the only instance to say yes or no if thats not enough
zealous gazelle
What's the email of GGG? xD
simple ravine
I'm sure you'll find it on their website.
worthy cape
@zealous gazelle You could look at prior art like Procurement and Exilence, which runs on a client computer, interfacing with only the stash tab API where it gathers information about their stash contents.
GGG isn't overly keen on people sticking session IDs into tools though as it grants a somewhat unmitigated amount of power over the user account.
It's the oldest kind of tool in the book, pretty much.
long fiber
Does anyone happen to have a detailed reference, or better yet a sample Python script, on how to query the API? I am so confused it seems I can't even get the most basic request thru, and now I can't find the one good reddit post detailing it which I came across last time I tried to play with this..
rapid pagoda
Which API?
long fiber
Is something like this supposed to work? :
search = {'query':{'status':{'option':'online'},'stats':[{'type':'and','filters':[]}]},'sort':{'price':'asc'}} # Some query string
url = 'https://www.pathofexile.com/api/trade/exchange/Standard'
r = requests.post(url=url, json=search)```https
long fiber
I wrote that one months ago so not sure anymore if the search string is valid, just curious on the structure if it's supposed to be OK
Well I mean, clearly something isn't working since I'm getting status_code 400 but I'm not sure where (even with https)
rapid pagoda
A search for an item by its stats doesn't make sense for the exchange search
worthy cape
Exchange is bulk items?
long fiber
Hmm didn't realize that, tho it makes sense since it's just the stream
(to my understanding)
rapid pagoda
The stash tab "river" is a completely separate API from the trade search API you're using. The river can't be filtered, and isn't league-specific
long fiber
I'm confused what is it I am using then?
https://www.pathofexile.com/api/public-stash-tabs = river
https://www.pathofexile.com/api/trade/exchange/Standard = ..? listed items?
rapid pagoda
First one is the river. Second one is the bulk item exchange search which backs the web page https://www.pathofexile.com/trade/exchange/Standard
The sort of request you'd send to that endpoint would be something like
{"exchange":{"status":{"option":"online"},"have":["chaos"],"want":["alt"]}}
Ha that yields a 200 now we're talking
rapid pagoda
If you wanted to search for normal items like gloves or whatnot, the endpoint for that is https://www.pathofexile.com/api/trade/search/Standard (no /exchange/)
long fiber
Sweet
And is this the best resource for figuring out query keywords? https://pathofexile.gamepedia.com/Public_stash_tab_API
Maybe it's because I'm a beginner but I feel like the whole API has so little documentation it all has to be reverse engineered and is dependent on the good will of third party devs..
rapid pagoda
@long fiber No, that's for parsing output from the river. For the trade site, best "resource" is forming queries from the web interface and looking at the requests that generates
long fiber
Yess!!! That first one is the one I looked for so much x_x
rapid pagoda
The trade site is kind of an unofficial API. It primarily exists to make the trade web site work, not for other clients to consume
long fiber
Ok sweet I'll play with that, I had seen that mentioned in an old post but thought there was a better documented alternative then reverse-engineering from the network tab
Love you both so much
I'll store these information safely as reference should be good to pursue my tinkering (..maybe) โค๏ธ
Thanks again it somehow seems so clear now and I manage to get some of what I want done
rugged pecan
would this be a place to talk about vulcan?
violet path
that depends on the question so choose carefully
rugged pecan
cause it seems to be having issues with screen shots
when hovering over an item it shows stats but if you use print screen and copy paste to say art its not showing the stats popup u get hovering over it but both chat and any infleunces will change on the current screen shot
worthy cape
I'd recommend spelling it correctly as "Vulkan" - earns you nerd cred.
rugged pecan
sorry
worthy cape
(also, this is not the place for end-user problems)
worthy cape
No, no. Read me right ๐
rugged pecan
i see
worthy cape
This channel is for developers talking about developer:y things with each other, if you've got a problem with something in the game, use the bug report forum or the techsupport email.
Note that the game has its own screenshot key at F8.
(or F12 if it's Steam)
rugged pecan
sorry ill leave now
long fiber
Tbf this discord could do with a "Technical issue question" subchannel (not that this one should fill that gap, but who has time for waiting after moderation/techsupport anyway)
worthy cape
@long fiber Normally we point at #1โendgame-help
long fiber
Yeah seems like the only applicable one
simple ravine
Complaining about features in beta, is like complaining like an entitled child in open source github issues ๐
worthy cape
@simple ravine There's a difference between inquiring about a problem and complaining, this person was rather courteous.
tacit birch
Saw that the poe ninja data dump has a typo ^^ The delirium section is called Metamorphl, don't know if it' the right place to call this stuff out
zealous gazelle
@worthy cape I emailed GGG about using the stash tab API and they said that any API that's not under the publicly available resources tab is not supported.
So I had to discontinue my tool.
wait really?
oh wow,
that just made me happy, thanks
worthy cape
Don't forget to set a nice user agent and mind the rate limits.
zealous gazelle
wdym?
I might have a problem understanding what "obfuscating your user agents" mean, care to explain?
oh I see! thanks,
worthy cape
In HTTP requests, you get to identify what kind of "user agent" you are (browser brand, version, tool, etc.)
It's encouraged that you provide the name/version of your tool, and maybe contact details, so that GGG can get hold of you if you're doing something naughty.
worthy cape
Web standard for delegating authorization, in the PoE case it lets an user authorize websites and tools to access some of your account.
An example is that filterblade can have you authenticate your account via OAuth and can then sync loot filters to your PoE installations.
A huge security benefit is that the tool/site never gets hold of your details, it just forwards you to GGG's OAuth infra and gets a token back with which it can use to act on your behalf for the purposes stated.
Compared to how you have to diddle with yoinking session IDs out of browsers or give tools your account credentials.
zealous gazelle
that's nice
obtuse citrus
it's 0
aren't you using python anyway?
if so you should just using the pypoe functions for it instead of reinvesting the wheel
worthy cape
Felt like cheating the other day using PyPoE and RePoE, too easy ๐
obtuse citrus
well I'd parse what you need then create your own json file
dumping dictionaries to json is trivial
to get the text
they're not
Metadata/StatDescriptions/*.txt
you can shortcut the processing via
it functions like this
>>> tr.lines
['+(8-12) to Strength']```see rather then going the repoe files in json I'd assemble the things you need in python with pypoe since it's designed around working with the raw files
['Adds (6-7) to (9-10) Fire Damage', 'Adds (6-7) to (9-10) Cold Damage']
r is RelationalReader instance and tc is TranslationFileCache instance
Just play around in idle
it's old but should still work I beleive
zealous gazelle
@worthy cape Hey man, it's me again. Do you remember when you said "Not supported doesn't mean it's not allowed" . Can I ask where this info comes from? Because I don't want to get banned for my tool, and I also don't want other people to get banned either.
worthy cape
@zealous gazelle that was xanthics ^^
simple ravine
Did you not find the support e-mail on their website?
simple ravine
then I assume you asked the wrong question.
and asking a different (the right one?) here
zealous gazelle
here is what I asked:
"So I wanted to ask if using this API is allowed or not even though it's unlisted? "
worthy cape
I can understand the reluctance and can only point to prior art.
zealous gazelle
I haven't asked it any other way , this has always been the main question
simple ravine
Ask outright: "If I am creating a program with X, Y, Z functionality. Will it result in people being banned?"
zealous gazelle
Alright then, I don't know how my question is not straightforward, but I will try and ask it your way then.
simple ravine
Just trying to help, as it seems the question you asked did not satisfy what you wanted
Nor did any of the people's insights here, so I dunno what else there is to do
zealous gazelle
you are right, there might have been a misunderstanding.
here is what I just enquired via email:
"If I am to create a tool that lets players download their own stash tab's content for the purpose of extracting data from their HortiCrafting stations via a button will I and the users get banned for it?"
I can't be more clear than this.
grave wren
yes and now they can check context too
cosmic saffron
make sure you specify it doesn't interact with the game client
zealous gazelle
it doesn't,
long fiber
Is it possible to retrieve unique items' roll ranges from the API or you'd have to scrape the wiki or poedb?
I know the one for uniques names and bases but I'd need to have their values
carmine merlin
@velvet fog have you thought of including the ailment threshold numbers for mobs? https://www.reddit.com/r/pathofexile/comments/fbkzx0/ggg_why_cant_we_get_the_numbers_needed_to/fj5vo33?utm_source=share&utm_medium=web2x
zealous gazelle
"Our ToS clearly states that we reserve the right to terminate any account that shares their login information with a third party, including their session id." Okey, this is pretty clear.
Looks like I'm terminating my project then.
broken cloud
@zealous gazelle It looks like you want a clear "yes, your project is OK" answer from GGG... that's your call if you want to play it 100% safe, but from experience, they will go out of their way to avoid saying this explicitly.
Third-party PoE tools that require the user to input some kind of credentials have existed for years, and GGG is aware of their existence (especially now that you have to identify your tool in the user-agent header), and no one has been banned for them.
zealous gazelle
this is absurd,
I have started looking into other ways I can get stash tab contents, and I just can't find any
people suggested OAuth, but I can't find any source on how to use that.
broken cloud
There's a GGG guy in here talking with us tool developers regularly (hi Novynn!) - he's the one you want to talk to for OAuth stuff
zealous gazelle
Like honestly, if I were to start using my tool as it is currently, and some day they decide to just ban users , I don't want that. I would rather avoid the risks, unlike POE overlay
Novynn you say? Do you think I can bother him with a direct message or does he have an email?
broken cloud
I'd go with email, roryv@grindinggear dot com
zealous gazelle
thank you so much.
worthy cape
@broken cloud What kind of workflow would one have with a purely client-based tool and OAuth?
Like, do you pop out into a browser and somehow get things back into the app? Seems sketchy if you don't have any site involved.
(I'm quite web-incompetent)
zealous gazelle
OAuth is probably just a middle-man between your app and pathofexile.com,
if I understand it correctly,
worthy cape
On a website you have some faith in that the browser isn't compromised and won't exfiltrate credentials and honors cross-site junk. Most of that is lost in a freestanding app.
zealous gazelle
So you think that if I don't make my tool web based I don't have a chance for getting access to OAuth ?
worthy cape
I have no idea, just thinking aloud ๐
broken cloud
@worthy cape Not sure yet, haven't gotten around to actually trying it out... but this seems to imply that a browser is not required https://aaronparecki.com/oauth-2-simplified/#others
zealous gazelle
@broken cloud that email doesn't exist ๐ **Edit nevermind my bad, forgot the .com
obtuse citrus
did you read the docs?
civic hawk
Hi. I got a little bit of an unordinary question...
Basically I'm looking for a VS Code theme that looks like the one from picrel. It's from the MonoLisa website and even tho it is stated to be a Winter is Coming theme it looks nothing like it. I really like this one so maybe some of you knows a theme that resambles it even a little bit so I can edit it without spending 10s of hours on it ๐
https://cdn.discordapp.com/attachments/588425873859018762/731904190296621097/unknown.png
I found a couple similar ones like edge ocean, but theyre pretty hard to read
this one however its just perfect for me
indigo relic
https://github.com/TheBelgarion/PoeHarvestList just simplified an other tool coz i just wanted a list with all my mods without pricing and so on
quick basin
hey guys, would it be against the ToS to have a tool automate chat (keyboard) input for you?
via send keys
worthy cape
One action needs to do a specific fixed reaction.
You can't have a key do different things based on some circumstances, or cycle through different actions, but you can do things like "hit F5 to type /hideout<CR>"
quick basin
yeah i was gonna use it to do open chat and squelch a bot
but i dont know how that is different than say, doing some flasks
its more than 1 key either way
rapid pagoda
Since that isn't a true in-game action, that's probably fine, but be aware that your ignore list has a limited length
quick basin
heh, just for testing now
was gonna squelch all the bot accounts
see how far it goes
just dont want to get banned lol
hazy wigeon
Created this simple AHK script to simplify the pricing of horticrafting stations to be parsed by forbidden harvest
https://github.com/SteffenBlake/FarmersMarket
Cheers!
It just copies the item text, then modifies your clipboard to now hold the properly formatted note that can be parsed by forbidden harvest stuff, you have to still set the note yourself so it doesnt violate GGGs user agreement and etc, it doesnt do any actual actions, just modifies your clipboard is all
simple ravine
Setting the price should be fine as well. The rule of thumb is 1 server action per key-press, @hazy wigeon
hazy wigeon
Thats what I thought, ok so now for the important question:
Is Ctrl+C a server side action?
simple ravine
no
hazy wigeon
Or can I do the entire process of:
- Ctrl + C (Maybe not server side action?)
- Parse and modify the clipboard (definitely not server side)
- Right click an item in stas
- Paste into the note
- Close and set the note
Nice thats what I was hopinh
Ok so, now I guess what I think I am gonna do is create a strawpoll or something for people to request what their ideal keybind would be for the action
grave wren
Or make it configurable with a reasonable default
hazy wigeon
I have to learn autohotkey's UI stuff on how to make a menu you can edit and etc
simple ravine
Config file for first iteration prehaps
worthy cape
Alt is a sketchy key, as it sometimes catches AltGr, which many keyboard layouts use for regional characters.
hazy wigeon
hey theres a command to go to your grove right?
I think I might add a hotkey as well you can tap, say F10 or whatever, to execute that command and warp to your grove
Same functionality as F5 on the trade tools
simple ravine
For the .NET developers out there, the performance improvements continue to impress... Check out .NET 5 perf improvement blog post: https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-5/
quick basin
Dotnet is awesome
dry atlas
Hey, I've recently switched over to SSF which means I don't have to keep updating things like trade macros, but there's one function I really liked about it which is the instant /hideout button. I have zero coding knowledge but I'm literate and good at googling. I made this
F5::Hideout() ;
F6::Harvest() ;Hideout(){
BlockInput On
Send {Enter}
Sleep 2
Send /hideout
Send {Enter}
BlockInput Off
return
}Harvest(){
BlockInput On
Send {Enter}
Sleep 2
Send /harvest
Send {Enter}
Sleep, 75
BlockInput Off
return
}
it works, but it displays the macro typing it in chat and also displays an error message if it can't do it
is it possible to do it in the way that TradeMacro does it? it's instant and doesn't show it typing
worthy cape
Look at its source?
mortal bone
SendInput {Enter}/hideout{Enter}
thx
mortal bone
You should almost always use SendInput or SendPlay since they buffer keys and don't allow user input in-between
dry atlas
I see
slim heath
Is any free to perhaps answer a question about Trademacro for me?
simple ravine
@slim heath you're looking for #1โendgame-help
slim heath
Sorry, was directed here... I will ask there
quick basin
iirc, a single key is allowed to do 1 action towards the server?
rotund solarBOT
GGG is very strict about macros and automation. Your macro must follow this rule: a button must perform only one server action and it must perform that one single server action every time with no condition. If you have questions about what counts you are welcome to ask but questions about what GGG can detect are not acceptable. Please do not argue about what other games allow, GGG has their own rules.
quick basin
damn
that means i cant really do: set tons of spammers to ignore with one keypress
๐ฆ
yeah but im executing the ignore action within the game
i got the producer, but the whole point is for people to be able to press 1 key
pressing tons of keys is not much better than typing it yourself
on the upside, i can still do the "single ignore" flow. Parse last 16 lines, if spammer found -> ignore him
mortal bone
I think this might be a bit of a grey area, but they really don't like queued systems either. For example, you can't cycle through your flasks with one key press. I am not sure if repeating the same action with different inputs would fit under this.
simple ravine
certainly grey area, but the use case doesn't give you tangible competitive benefits, so might look past it in this case, but yeah..
worthy cape
An UI more like MercuryTrade where you have an interactable UI element per bloke is a bit more in line with the spirit of things.
rapid pagoda
The sense I get is that GGG cares much more about tools which automate gameplay actions (like using flasks or abilities) than ones which automate OOC actions (like chat or trade)
Which isn't to say that heavy automation of either is okay, but you'll get a lot more leeway for something like an auto-ignore macro than something like a flask or ability macro.
rapid pike
We got a list of every single craft from Harvest?
worthy cape
@rapid pike In an early announcement - https://www.pathofexile.com/forum/view-thread/2874544
worthy cape
No clue.
rapid pike
Haha, no worries.
It beats guessing, for sure.
Any guide on using the API?
I would like to receive stash tab data.
worthy cape
There might be some more definitive tables in Data, you might want to check if PyPoE (or RePoE) has some specs.
rapid pike
Is there any issue with me using AHK?
worthy cape
@rapid pike As in the public stream of listed stash tabs from the river? There may not be information on Horticraft stations in there, if that's what you're after.
rapid pike
How does TFT index them?
worthy cape
Ah, the crafts are in the river.
It's just the trade site that doesn't expose them.
rapid pike
And out of curiosity, how do I grab data from the river using AHK?
Is there a guide on that?
worthy cape
You make a request to the river endpoint with a change-id. The payload is a big honking JSON object with information on stash tab changes since the change-id. It also contains the next change-id for you to chain with.
There's a fair bit of information in the pinned messages.
rapid pike
Alright. I'll take a look into it.
worthy cape
Note that this is a hefty amount of data and something you'd process continuously, sifting out the information you care about.
It came in at something like 5 MiB/s when I poked at it the other league.
worthy cape
When a public stash tab is modified, all the contents of it are send out anew.
rapid pike
and then auto-generate a csv
worthy cape
If you want to query a particular user's tabs, there's different authenticated interfaces for that.
rapid pike
Is it okay to do a macro that sends a message to the game incrementing a number each time?
Like, /trade 1
/trade 2
etc
so i can spam easier lmao
worthy cape
No.
rapid pike
Fair enough
But i presume mapping it to some kind of crazy combo for like CTRL + ALT + Z + 1 for trade 1
and then going on through the letters
Would that be fair game?
Meh, it's not a big deal i just get carpal tunnel trading in this game
AH when
worthy cape
Sounds more reasonable, but I'd recommend not trying to force a workflow that's not fit for the game and the rules.
rapid pike
Yeah probably best to focus on something else
rapid pike
https://docs.google.com/spreadsheets/d/1ZsQhkQVTPDB0AWodQIB4h19Cx5_4hf2N8wbWIIiiYv4/edit?usp=sharing
in case anyone wants them itemized
cyan root
does anyone know if there is an API for mod tags and the likes, just like on poedb?
idk where poedb gets the data from
worthy cape
The RePoE JSON files has a lot of the information you want, digested from the Data files of the GGPK.
There's earlier discussions in here how to navigate the maze of mod types, mod groups and all the other madness around this.
(I gave up ๐ )
cyan root
i was thinking of making a discord bot for it, but as always i need to start looking for data... ive never heard of RePoE before, i assume its not the official source?
and an official API for that doesnt exist?
worthy cape
Pretty much all the concrete information is from the client-side tables in the GGPK, that's the authoritative source. RePoE provides more user-friendly JSON files derived from those tables.
Depending on what you're doing, there might be something you can glean from the Path of Building source as well.
If you're looking to interact with the GGPK, PyPoE is rather friendly and what powers RePoE.
cyan root
oh, thats neat, i wasnt expecting a python module, thatll help
worthy cape
poedb is fed by the GGPK files and whatever dark magic the author applies.
cyan root
im just concerned about updates and stuff, i really wouldnt want to manually hand craft new mods and shit
and instead just plug a source for mods, but guess its not as easy
i assume someone, probably the poedb dev, already tried asking for an official API on mod tags and the likes, and it wasnt accepted?
worthy cape
There's not really any such data service at all, everything public-facing is baked to some extent.
Stats from mods providing the same thing are merged, etc.
cyan root
meh. thats disappointing for such a complex game with so much data. i guess ill check what the official API has to offer first...
worthy cape
So what will the bot do? We might have some advice on what is possible or not.
Note for example that items copied from the game with Ctrl-C have prebaked stats, so you can't always uniquely tell what mods there are on it.
cyan root
well basically solve the endless "how do i craft x" "will this be 100% save crit chance" questions
type some command with keywords like "HelmetDexInt_Shaper_Crit" and itll spew out all mods with those tags. thats the rough idea
obtuse citrus
i assume someone, probably the poedb dev, already tried asking for an official API on mod tags and the likes, and it wasnt accepted?
I actually asked about this a long time ago and it wasn't really something they were planning to do
Some people at GGG would rather not have some info available as well. Some others are more open to it
I can tell that they'd rather not have a number of things available since there have been steps taking to remove certain data entirely (say mod sell prices and various league mechanics depending), and they do things like removing data from the pre-release patch and obfuscate the unique item stuff
Though no idea what the official stance on this is right now
I remember weighting data for instance specifically mentioned that they might remove it entirely, but that was many years ago and they haven't, so ... no clue
worthy cape
I wonder if PoE 2 will have some big change in some direction.
simple ravine
oh I hope so, that will be fun xD
worthy cape
Heh, my GGPK fetch project was halted by an unexpected factor today.
Accidentally pulled out the DAC SFP+ from the server with the vacuum cleaner ๐
simple ravine
what's a DAC SFP+
worthy cape
SFP+ are the modules you stick into a 10 Gbit NIC with the optics.
A DAC is a cable with two SFPs hardwired on copper on both ends, a Direct-Attach-Cable.
simple ravine
Oh, I think I have two SFP+ ports on my router
Never had the pleasure to use it though
You have 10 Gbit at home?
You're worse than me! ๐
worthy cape
My super overkill core switch
Internal networking is 10gig or 1gig, internet is currently bumped up to 300/100, normally 100/100.
simple ravine
I'm actually considering downgrading my internet speed
worthy cape
I upgraded to 300 for the PoE download project - turns out that Steam servers won't give me much more than 3-6 MiB/s with the current tooling ๐
obtuse citrus
My ISP offers 1 gbit to new customers for the same price I pay now (permanently), but would charge me about 50% more if I wanted to upgrade from 200 mbit
:/
worthy cape
Hey, you people probably know more SQL than I do... I've got this table relating releases with each other:
CREATE TABLE ordering (old INTEGER NOT NULL, new INTEGER NOT NULL, UNIQUE (old, new));
simple ravine
worthy cape
It kind of forms a linear list of before-after relations, and I want to have queries like "give me all versions between release A and B".
simple ravine
It's so loud I need to keep it in my closet
worthy cape
Never quite managed to wrap my head around CTEs and recursive queries, is there any layout that would answer my question better you think?
Right now I just suck it up and issue N queries in sequence from bash ๐
simple ravine
I havent written a CTE query in a very long time
worthy cape
@simple ravine One of the big factors for my Mikrotik switch was that while it has fans, during normal home operation they're not active and the passive heatsinks suffice.
simple ravine
and then it was SQL Server, not sure what you're using
worthy cape
sqlite3
simple ravine
maybe a reason for you to whip out Neo4j ๐
worthy cape
Stared a bit at their docs for CTEs and with recursive and decided that I'm not a DBA ^^
simple ravine
yeah, I've only used MySQL and SQL Server 'professionally'
I want to get more accustomed to Postres... though, SQLite isn't really that interesting for my use-cases
except hobby stuff like poe side projects
re router, I've seen people switch the fans out for 4" Noctua fans and it becomes really quiet
Ubiquiti one goes full force no matter what
worthy cape
I'd like to share one of last night's bugs:
overlapped.Offset = off & 0xFFFFFFFF;
overlapped.OffsetHigh = off > 32;
I promise I'm a competent programmer 
simple ravine
bitshifting ๐คข
worthy cape
Win32 OVERLAPPED takes the offset split into two DWORDs because of reasons.
worthy cape
Manifested in one of two different ways, depending on the size of the file. Either bailed on the bounds check on the read or silently read in the wrong place.
simple ravine
You will have to tell me what the issue is
worthy cape
@simple ravine The intent is to shift the high 32 bits of the offset down and assign it to the high dword. The code currently does a greater-than test and implicitly converts the bool to 1 or 0.
C# got rid of that bullshit conversion, right?
simple ravine
I think you need to state quite explicitly what you want nowadays
But I haven't done low-level programming in a while, so memory is a little fuzzy
aside from that @worthy cape, I thought Windows didn't have Overlapped IO... only Linux?
worthy cape
Async I/O in Windows is ancient.
simple ravine
Right, it's done using regular completion ports if I am not mistaken
worthy cape
Open the file with the right flags in CreateFile, prepare an OVERLAPPED struct with an event to signal or associate it with some IOCP queue, ReadFile to your heart's content.
There's other async I/O functionality too like the one introduced in Vista which I can never remember the name of.
simple ravine
ah, i just googled it
worthy cape
Linux users seem rather happy about io_uring nowadays, unlike the previous madness that was epoll and friends.
simple ravine
turns out my memory is a dumpster fire
worthy cape
See, that what happens when you use managed memory for too long.
simple ravine
๐
It's all coming back to me now
another thing that works better in Linux, that I hope they "fix" in Window, is how memory mapped files are read and allocated
if you're curious, try memmap Content.ggpk and go into Task manager.
it "looks" fine, but then go check allocations
I was able to read Content.ggpk structure really fast, but I got a little scared
the new way I'm doing it is quite fast anyways
worthy cape
What's the matter, high virtual size?
simple ravine
yeah
well private allocations or whatever it's called
said like 12 GB
so not virtual as in paging file
worthy cape
They're still backed by their file and on the standby list, so they can be scrubbed and evicted when paging pressure dictates.
worthy cape
The memory manager aims to have a suitable amount of zeroed pages and will consume from the tiered standby list when needed.
simple ravine
perhaps something for a night's discussion
worthy cape
Pretty similar to the file cache, but a bit more explicit.
simple ravine
gotcha
So you'd say that even though there's reported tonnes of allocation, it's no sweat?
worthy cape
If you're using Process Explorer, the memory tab shows the sizes of all the paging lists btw.
simple ravine
My problem was, if it was used by a computer with lower-than the file size in memory
worthy cape
Unused memory is wasted memory, you're only in trouble if you've got enough pressure on RAM.
On a machine with less memory, you'd have fewer pages resident.
When running your program, you might evict pages of other processes as it's hard for the manager to figure out if you only touch pages once or intend to revisit them.
That's one benefit of explicit I/O, you typically have your short-lived read buffers.
simple ravine
Well, if I allocate 30 GB in my app explicitly, it's not up for grabs
worthy cape
If you slurp in the whole file, you're in a worse place than memory mapping indeed.
As then there's only the page file backing you for eviction.
simple ravine
Yeah, I think there is a way to have a 'window' where you tell the memory map what part you're interested in, and move that window around
worthy cape
I mean the typical stream-based I/O you'd do with traditional files, you only ever have a few megs of buffers typically and consume the result.
simple ravine
But then I got lazy
worthy cape
You can indeed map views of a file and be more careful about what you touch, but too many people treat memory mapped I/O as "map the whole thing and just point all over the place".
simple ravine
That's what I tried for an hour or so a couple of years ago
worthy cape
One of the curses of x64 ^_^
worthy cape
One of the reasons why I went with overlapped I/O for my current incarnation of GGPK reading, as I only need one file handle I can share freely between threads.
simple ravine
the code you showed me looked very c# like
in Managed C# you can create a managed async file handle
worthy cape
I'm trying to re-skill (upskill?) in C++ again.
simple ravine
ahh ok
worthy cape
Lots of weird habits from Rust now, and lots of old aversions to train away.
simple ravine
I have no desire to touch C++
worthy cape
Like an allergy to exposing any form of implementation detail members or platform stuff in public classes.
simple ravine
Oh that reminds me
even more hardware intrinsics in .NET 5
tons of it in .NET Core 3+ already
not sure if it's worth spending time optimizing GGPK reading etc though hehe
just extract what you need and get out of it imo
main meteor
Is there a program that checks all items on your account, characters too?
worthy cape
"checks"?
main meteor
lets me search
worthy cape
Procurement did, I believe.
quick basin
guys can someone send me his poe logfile? (in case they have not cleaned it for a while)
i need a ton of chat messages history
golden bane
That's pretty private information you're asking for
worthy cape
Even for well-known figures in the tools community, I wouldn't share mine without going through and removing PMs, guild, and probably global, and that's not exactly relevant then ๐
quick basin
sorry, didnt assume theres private stuff there
for me its just vulkan stuff and talks abouts seeds lol
mortal bone
I have log files all the way back to 2017 haha
The first and last entries in my log file lol
2017/05/24 17:47:51 ***** LOG FILE OPENING *****
2017/05/24 17:47:52 2796984 148 [INFO Client 4104] Send patching protocol version 5
2017/05/24 17:47:52 2797046 1b7 [INFO Client 4104] Web root: http://patchcdn.pathofexile.com/2.6.0.16/
2017/05/24 17:47:52 2797046 1b8 [INFO Client 4104] Backup Web root: http://patch902.pathofexile.com:8095/patch/2.6.0.16/
2017/05/24 17:47:52 2797046 1c3 [INFO Client 4104] Requesting root contents 1
...
2020/07/16 07:16:19 ***** LOG FILE OPENING *****
2020/07/16 07:16:20 ***** LOG FILE OPENING *****
2020/07/16 07:16:21 23012093 1c7 [INFO Client 15276] Send patching protocol version 5
2020/07/16 07:16:21 23012234 23d [INFO Client 15276] Web root: http://patchcdn.pathofexile.com/3.11.1.3/
2020/07/16 07:16:21 23012234 23e [INFO Client 15276] Backup Web root: http://patchcdn.pathofexile.com/3.11.1.3/
I don't want to scrub 3 years worth of stuff
worthy cape
Time to bring out the profiler - 190s to verify a hot Content-1.0.0.ggpk
simple ravine
Verify what?
worthy cape
Verify the SHA256 hashes of directories and files, ensure that the free list is valid, make sure all entries are well-formed.
I don't quite trust the downloader, so I wrote a dedicated verification tool that distrusts everything in the pack ๐
worthy cape
The actual depot thing is a third party project on github, I've just glued a ton of sqlite3 and bash around it ๐
simple ravine
Oh, so there's someone who already wrote a Content.ggpk downloader thing?
worthy cape
Plan is to grab everything, keep the first version in each league and make forward delta patches for all the intermediary versions.
And then also copy all the files from all packs into a single directory, deduplicated by naming them after their hashes, and leave small "outlined" GGPK files that don't have any file data in them.
Ultimate goal is to have a corpus of packs that I can easily swap between without wasting double-digit terabytes of disk space.
worthy cape
Chuan manually matched the major releases against particular download dates, so I've got the milestone releases.
I don't know if there's any reliable way of determining the version number of a PoE installation, or if it's all part of the client's communication.
I hope it's not in the EXEs, I deleted all of them last night ๐
worthy cape
It's not in the PE resources at least.
simple ravine
nope
the question is how much of the logic that's actually in the .exe vs other places
btw my ggpk file is 32 GB
wtf just happened lol
worthy cape
"Content"
mortal bone
Yeah, mine is 32 as well
worthy cape
Just shy of 36 GiB if you're a Steam pleb.
quick basin
btw about the logs, can I just get the messages with those annoying bot ad spams, LOVEPOE etc?
i need it to build a model
worthy cape
The string shown on the login screen is stored in the executable.
mortal bone
I don't turn on trade chat
quick basin
its not in the trade chat lol, thats the problem
they spam the global, so im trying to fight it
worthy cape
mortal bone
I also don't turn on global lol sorry
quick basin
haha ok np
mortal bone
The content.ggpk should have the version right? 3.11.1.3?
quick basin
its pretty odd to me that they dont logrotate btw
worthy cape
@mortal bone Hrm, where'd that be?
mortal bone
sorry, I was just saying that it should have the version, but they don't use the letters internally
simple ravine
I'm guessing that versions such as 3.11.3a is server side
And the current version communicates client content requirements?
mortal bone
you might be able to infer the version based on passive skill tree data...
It doesn't update frequently enough to do anything useful with it
worthy cape
Right now my priority is getting delta compression going, 300 GiB of headroom on the filesystem.
mortal bone
damn...
are you thinking about doing deltas between major versions?
from what I know everyone's ggpk layout is going to be slightly different in terms of space usage depending on when they started
worthy cape
I'm basing this on Steam GGPKs, so it's all stable.
mortal bone
oh i see
steel geyser
Hey guys,
i'm new here and i am trying to create a website for poe content.
Are there ways to authorize a player as such, by logging in to the PoE website?
steel geyser
yea exactly, or like poemap
worthy cape
You can find contact details in the channel pins.
worthy cape
Most trade:y things tend to be on some CDN, aren't they?
steel geyser
idk ยฏ_(ใ)_/ยฏ, would be cool to have access to it though
hopefully somebody here knows how it works
steel geyser
I've never worked with a cdn. How do i access its contents?
worthy cape
If you squint at the images on the trade site, they will look like https://web.poecdn.com/image/Art/2DItems/Currency/CurrencyVaal.png?v=64114709d67069cd665f8f1a918cd12a
There might be some info in the channel history about building them.
steel geyser
There has to be an easy way of looking through all the available content of the cdn right?
mortal wave
any clues which lib path of building uses for the gui? I'm glimpsing at the souce but I couldn't find anything
mortal bone
@mortal wave it is closed source iirc
mortal wave
their tool runs quite smooth actually, which led me to think it's not using lua
grave wren
It isn't it's the one DLL that's in there I think
golden bane
Over at PathOfBuildingCommunity, we've set up a repository with the latest version of SimpleGraphic.dll that Openarl made the source code available to the public. Currently there are several people working on setting up an automated build pipeline and fixing the long-lasting performance concerns. https://github.com/PathOfBuildingCommunity/PathOfBuilding-SimpleGraphic
grave wren
c/c++ shivers
steel geyser
are there APIs that list all crafting options?
mortal wave
the renderer really was written from scratch. ouch
I wonder if there was any reasoning behind that
steel geyser
@earnest radish thanks ๐ thats pretty overkill for me though xD
i realy just need bench crafts and maybe harvest, but primarily bench crafts as they are in vanila poe
the link from the brather1ng github should list everything i need, thanks! ๐
cosmic saffron
steel geyser
@cosmic saffron is there an api from poedb for the craftin options?
outside from SAP, i know nothing about the coding world outside lol.
How would i grab the json from the page?
simple ravine
hehehehe
step 1, pick a language
how did you learn programming in SAP, without learning proper programming?
steel geyser
they have their own tools for almost every solution, lots of proprietary stuff
simple ravine
I know. Most developers hate interfacing with SAP, because it's a dumpster fire
except the ones working with (inside of) it
steel geyser
hey they have tables instead of arrays, they are pretty lit and consistent with datatables
simple ravine
yeah, .NET has had them since 2000
steel geyser
i guess im working with sticks and stones lol
simple ravine
but anyways, pick a language
grave wren
(dont pick php :>)
steel geyser
i'm working with dart atm
simple ravine
dart is a frontend abstraction, so i guess you're not gonna do any backend?
Why not go with UI5? 
steel geyser
Is that some inside joke i'm too outside to understand
simple ravine
UI5 is some SAP garbage
grave wren
ui5 is a js framework from sap
simple ravine
SAP likes to re-invent the wheel, and claim they invented it
simple ravine
and in the same time, manage to make it worse than the original
steel geyser
dont all big software companies do that
simple ravine
no, not really
grave wren
negative
simple ravine
SAP: "we invented REST"
Everyone else: NOPE
steel geyser
pls dont make me re evaluate my future job decisions xD
simple ravine
perhaps the best thing that could happen today
specializing in a product like SAP is dangerous to your career happiness
i.e. Altavista vs. Google ca 1998
mortal wave
we like to bash php but literally all the big tech companies still use it
simple ravine
nope
mortal wave
and it's not legacy code
steel geyser
i just wanted to get the json from poedb and get demolished ๐ฆ
simple ravine
well, you kinda need a plan ๐
grave wren
php<7 is pretty much legacy
mortal wave
You guys have an invite to the pob discord?
simple ravine
I respect people writing PHP code.
But I still think PHP is subpar, and no, not every big tech company use it
grave wren
afaik its private
steel geyser
@simple ravine btw my backend would be firebase
simple ravine
that's your database.
your backend code
well I guess you can go directly from frontend to firebase... not sure I would do that though
steel geyser
hm its my first time working with web technology ๐ค
but afaik, firebase is not only a database, you can write your backend functions using node js
tardy widget
firebase is a cloud platform, they have a lot more than simply databases, it's basically just an abstraction layer over GCP
steel geyser
its a good platform, i just hate using NoSQL but i gotta roll with it
tardy widget
I believe firebase doesn't support SQL databases yet
steel geyser
firebase offers everything i need, hosting, databases, backend, ads
simple ravine
Google Spanner
tardy widget
tbh I've never used spanner, I tend to stay away from googles cloud offerings
simple ravine
the trollface should've given it away, it's a database starting at $600 / month
it's more geared towards enterprise grade systems
tardy widget
LMAO, yeah that's crazy, I didn't look at the pricing
simple ravine
Google has some good stuff, but I am not fan of Firebase
I think it's a swiss cheese disaster waiting to happen
tardy widget
Yeah I don't like that its so 'packaged' doesn't seem very flexible
simple ravine
Google bought Firebase IIRC
and then just blatantly put it on top of Google Compute Engine, and thought people would like that
I would never be able to pitch that to any of my customers
But their k8s stuff? sure
However, managed K8s has great support in Azure and AWS, so...
and they're more enterprise grade, in my opinion, especially Azure
tardy widget
Yeah you'd be better off just using GCP, sure the learning curve might be a bit more but you'd get way more out of it, and it's probably cheaper. I was looking at google K8's but once AWS come out with their offering I just use that, although I find it cheaper and better to just manage some EC2 instances rather than paying for EKS
simple ravine
In Azure, you don't pay premium for AKS (their managed kubernetes). You get the control plane for free
and with KEDA, it's pretty amazing how you can add burst capacity with their ACI (Azure Container Instances) without pre-provisioning anything
tardy widget
That's pretty sweet, I haven't really gotten far into Azure, I've been stuck on AWS for years and probably won't change for a while lol, but I might do some Azure stuff on some side projects
simple ravine
I'm Chief Cloud Architect at one of the bigger IT consultant corps, and most customers I talk with are our top 10 largest, i.e. big enterprise - so Azure is our biggest focus
suits me well coming from a developer background with focus on Microsoft & .NET since ca 2001
steel geyser
So which technologies should i use for my website ๐
golden bane
Is it a PoE related website?
steel geyser
yea of course
i want to create a platform for services to make them searchable and create a workflow using a queue-system.
Basically people join queues of services and service providers invite people from their queue
simple ravine
That will be a fun way to learn something outside your current work for sure
However, any answer to that question will be biased.
violet path
there was a website in the same vein as that for horticrafting that some of the tft people made, if you want to look into that for ideas
steel geyser
@jade adder i got that idea from animal crossing lol
simple ravine
I'd say there are 3 major things to consider:
- Node / JavaScript
- C#, .NET with ASP.NET Core
- Python
violet path
wrong jeff but I see you I see you
steel geyser
oh, sorry other Jeff
simple ravine
For sure there are other avenues like
- Elixir / Phoenix (quite advanced, difficult)
- PHP (perhaps not the best idea)
- Java (I wouldn't for the life of me)
steel geyser
boycott java?
steel geyser
let me show you ABAP
simple ravine
I've seen ABAP. Unfortunately.
They probably think they invented BASIC which it's based off of ๐
golden bane
@simple ravine made good suggestions though. You want to find a technology that is used by many people, well liked and has good third-party support basically
simple ravine
Either way there are a lot of programming paradigms that will help you become a proficient programmer.
- S.O.L.I.D. principles
-- Single Responsibility Principle
-- Open / Closed Principle
-- Liskov's Substitution Principle
-- Interface Separation Prinicple
-- Dependency Inversion Principle - Test Driven Design
- CAP and PACELC theorem
- Fallacies if distributed systems
- Domain Driven Design
- Command Query Responsibility Segregation
- Event Sourcing, and just events in general
That should keep you busy for a week or two ๐
steel geyser
Funily enough, (almost) none of that is being taught in my training, besides the usual suspects SOLID and TDD
simple ravine
Do you work as an SAP developer, or you in training for it now?
steel geyser
But i havent seen those principles applied anywhere
simple ravine
any other background of programming in school etc?
steel geyser
Just finished my training this month
simple ravine
University or?
steel geyser
starting to study soon
No university training, i was a trainee
It involved school training (basics in everything IT and programming)
That lasted for 3 years
Nothing realy practical, almost everything in theory, so everything I know is basically from SAP because the company i did the training in is an SAP partner
simple ravine
What kind of programming did u do during those 3 yrs?
steel geyser
95% ABAP 5% Java
simple ravine
in school?
simple ravine
steel geyser
๐ฆ
simple ravine
Well, one thing is for sure, some SAP consultants make serious dough
but I am not sure the ABAP programmers do that
simple ravine
But it's like time is standing still in that ecosystem
steel geyser
but starting at 45k actually
simple ravine
usd or euro?
steel geyser
โฌ
simple ravine
what country?
steel geyser
Germany
simple ravine
Ah of course it's Germany
steel geyser
Who else would work with SAP ยฏ_(ใ)_/ยฏ
simple ravine
The company I work for has the most amount of certified SAP consultants in the world
We're spread out over a number of countries
But it's not one of our top focus areas though
IIRC, it's 1. Cloud, 2. DCX, 3. FS
steel geyser
i dont know how many global players there are in the SAP consulting area
cuz i work at a company thats pretty much worldwide settled and does sap only
steel geyser
yep
simple ravine
Well, if it's SAP only, it's not the same company as mine
I work at Sogeti (part of Capgemini)
steel geyser
Never heard of them ^^'
simple ravine
I thought non of those companies did only SAP though
steel geyser
its hidden behind NTT
simple ravine
Only one I dont know much about is Abeam and NTT
Pretty much all other companies have had recruiters try to poach me though ๐
Sogeti/Capgemini = 270,000 people world wide. Interesting you haven't heard of them before.
steel geyser
i dont hear that much from competitors
i guess most sap consultants come from sap and switch to capgemini, or do they start their consulting career there?
simple ravine
no idea
but you also have to remember, a lot of sap consultants aren't programmers
but more business consultants
steel geyser
yea i know that
simple ravine
the expensive kind that knows the software module, and charge โฌ250/hr
steel geyser
been working with consultants a lot for custom development
they know the modules, we implement the ideas basically
steel geyser
@earnest radish your list of crafts is super helpful, thanks!
long fiber
Does anyone happen to have a uniques rolls table/db handy? Been working on one from poedb but it's a bit of a pickle
long fiber
Nevermind I almost got it, let me know if interested to have it
simple ravine
ouch, Cloudflare.
worthy cape
Veritania protects ๐
rapid pike
Anyone have a straight forward guide to pulling data from the poe trade stream?
I am trying to figure out how it works. Currently using AHK
worthy cape
Actual river now?
You're going to need something that can parse big hunks of JSON well, and act/persist the data well in a long-running program.
You make a request to the river stash tab endpoint with a change ID corresponding to some point in time. That will get you the oldest changes after that timepoint and the next change ID. You keep doing that (adhering to rate limits) until you catch up, and then occasionally poll for more.
rapid pike
Probably :<
I just dont know what else to use
My experience is in UE4, C++ and Godot
simple ravine
Well, that depends on what you already know
rapid pike
And I don't know how to make UI stuff in any of those
simple ravine
I have no idea what Godot is, but C++ can do the job
rapid pike
At least, I could learn, but it would take forever.
hazy fiber
Godot is a game engine, not a language
rapid pike
Obviously.
simple ravine
Forever seems like a exacerbation, but sure
rapid pike
But it still lets you interface with HTTP requests
hazy fiber
also both UE4 and Godot have pretty good UI tools
rapid pike
I'm not a fan of controls in Godot.
simple ravine
Why not use something that is more geared towards what you want to do, like C# or Java?
rapid pike
And UE4 UI tools are great but require a lot of setting up
worthy cape
The only way to access other people's stashes on demand is by squinting at the river.
rapid pike
I'm simply not familiar with them that's the only reason why I haven't considered them.
grave wren
time to gather new knowledge \o/
simple ravine
Welp ยฏ_(ใ)_/ยฏ
worthy cape
@simple ravine I have some exciting performance numbers... https://i.imgur.com/QDQQijz.png
validate_ggpk.exe on the current release, my old foe "I/O is hard".
worthy cape
Time spent waiting for the event associated with the overlapped I/O that I resolve directly.
simple ravine
SHA256?
worthy cape
ReadFile(h, p + i, static_cast<DWORD>(block_n), nullptr, &overlapped);
DWORD block_read = 0;
GetOverlappedResult(h, &overlapped, &block_read, TRUE);
Hashing is rather free, leveraging Windows' crypto providers.
simple ravine
ooh right, you're validating every block entry?
worthy cape
Yeah, traversing the whole directory tree, ensuring that each chunk is well-formed and compute all the SHA256 digests.
I want to make sure that my downloads aren't malformed in some way.
simple ravine
the way I did it, is following the rabbit like alice, which ignores the free blocks
and it was pretty fast
worthy cape
Free blocks account for like 0.7% of the CPU time.
worthy cape
I've got some scaffolding to experiment and tune in now at least.
Just a bit sad that VTune can't offer any insights on I/O.
simple ravine
well it was at least faster if you knew what "path" you wanted to go to, i.e. the Data folder and grab the dat files
much faster actually
worthy cape
I could instrument my file reading a bit I guess, see if I can tell if I do dumb stuff somewhere.
Sure, individual accesses are very lazy and snappy. It's traversing everything that's death by a thousand cuts.
simple ravine
it could be faster actually, to spin up N actual threads, and do synchronous file reads
async adds a little overhead
worthy cape
I've got buffering in place to reduce the amounts of syscalls done, but I feel like I should have an I/O thread to schedule reads.
simple ravine
where N = number of cores
worthy cape
I'm working against a single HDD here too, so concurrent reads would be unhealthy.
simple ravine
ah ok
worthy cape
The corpus is currently 3.5 TiB ^_^
simple ravine
yeah then the way I would do it is something like...
worthy cape
Ain't nobody got SSDs for that.
simple ravine
1 thread that delegates work to N-1 other threads
(where N is number of cores)
or something like that
optimizing for IO is not fun
but try messing with buffer sizes
worthy cape
I wonder if this tool would be better off with one upfront traversal of the directory structure and then hashing everything in file order.
simple ravine
well, because it's HDD, linear workflow is to be preferred right?
random access will decrease performance
so you'd have to sort by offsets then
it was a long time ago since I worked with the GGPK format, but the 'table of content' is at top right?
or is it mixed with data?
worthy cape
GGPK first holds offsets to FREE and root PDIR.
But all the PDIRs and FILEs are at wildly arbitrary locations.
So reading the structure is a costly scattered operation.
Thanks for rubber ducking, gonna play with the ordered scan later.
๐ฆ
simple ravine
reading my ggpk code to refresh my memory
If it's to any use, here's the code: https://github.com/andreandersen/PoeSharp/tree/new/src/PoeSharp.Filetypes/Ggpk
Note the branch new not master.
worthy cape
Implemented a two-pass algorithm that traverses the tree to gather all the file entries, hashes them in file order, and then traverses the tree again to verify. 72s ๐
simple ravine
what was your old time, @worthy cape?
worthy cape
Varies, but 900-1550s.
worthy cape
Fast enough for validating things on the server at least, which was what I needed it for.
mortal bone
72s isn't bad at all
buoyant scaffold
@timid hemlock Wanting to add Scinteel Synthete (https://poedb.tw/us/mon.php?n=Scinteel+synthete) to Spectres.lua & Skills/spectre.lua but not seeing where baseEffectiveness or incrementalEffectiveness are being calced.
Is there any logistical reason that poedb.tw mon.php (or pick auth source of data of your choice) isn't being scraped for Spectre=Y values to generate Spectres.lua instead of hard coding for each?
worthy cape
Bleh, timing on Linux is still bad, at least over NFS.
worthy cape
@simple ravine I need faster storage, got any spare flash SAN?
worthy cape
Source is 4x8TB disks in mirror, target is a single 8TB disk.
simple ravine
ah
patience is a virtue, my friend
interesting how it looks like u had half the speed for some time
worthy cape
I was computing deltas on the server at the same time there.
worthy cape
I'm making a backup of the data now before I start going all-in with deltas and only store milestone files.
Cloud isn't free, this is 4.4T of data right now.
I got a great idea about computing deltas the other day by the way.
Initial plan was to start with the first pack of each minor release and make forward deltas.
It'll probably be smarter to start with the final pack in a release series and make backwards patches.
simple ravine
Azure Blob Archive for 5TB is actually fairly cheap
$5 / mo
but yeah, if u want IOPS its not cheap
people saying cloud storage is cheap hasn't stored a lot of data in the cloud
it's one thing to dump data there, but it's another if u wanna use that data
worthy cape
Kudos to Azure, their pricing is easy to find and read.
Tried understanding AWS and GCP pricing recently for a project, utterly impossible.
simple ravine
https://azure.microsoft.com/en-us/pricing/calculator/ makes it easy
Makes sense to probably have Hot + Archive
velvet fog
@buoyant scaffold It's possible to provide Spectres.lua from web, if I have time
based on https://www.reddit.com/r/pathofexile/comments/hoip63/guide_to_manually_adding_spectres_to_pob/
worthy cape
@mortal bone Started grabbing EXEs as well as GGPKs now, ran into my first identical version tag:
zao@mim:/media/stuff/depots$ strings downloading/5755113008739565307/PathOfExileSteam.exe | grep 'release tags'
release tags/2.4.0b
zao@mim:/media/stuff/depots$ strings downloading/7962219399067405571/PathOfExileSteam.exe | grep 'release tags'
release tags/2.4.0b
Looking at patch note history, the second is likely a hotfix.
Good enough to orientate myself at least, but not unambigious.
simple ravine
Did you find a way to detect the version via GGPK, or are they decoupled in the sense of versioning?
worthy cape
Haven't seen anything immediately obvious, but haven't dug more than looking at things in pypoe_ui.
velvet fog
seems no way, there was a patchnote.rtf inside ggpk, but removed in newer version
simple ravine
I see there's a hashcache.dat outside ggpk - chuan u know on top of your head if that is usable?
velvet fog
no idea
simple ravine
ok, gonna check if omega has it defined in pypoe - nope
my theory is that they're disconnected. server will say "in order to run current version, which is x.y.z, you need the following files" or something like that?
worthy cape
The patch CDN has five-part version numbers - cf. https://i.imgur.com/ULgaKgO.png
Could've sworn that I'd seen that protocol described somewhere.
velvet fog
maybe they scan the PDIR sha256 to check if file was changed
HashCache.dat
typedef struct {
int len;
char filename[len*2];
char unk1[12];
char sha265[32];
} BLOCK <optimize=false>;
int total;
BLOCK block[total];
I can't find any useful information in HashCache.dat
simple ravine
char sha265 <-- my ocd, argh
velvet fog
according the logs/client.txt, they could compare PDIR to check if anything was changed under this directory
simple ravine
last time I was looking at PyPoE... a long time ago, I think there was an embryo of a downloader from the patch server
which I think was also checking the version somehow(?)
worthy cape
Saved like 1TB for 1.0.0-2.3.x with deltas by the way.
simple ravine
nice
mortal bone
@worthy cape That sucks, but I imagine that will happen a bit. I recall them deploying a patch and rolling it back as well then later deploying that same patch with fixes. There was a bug where no one could move haha
buoyant scaffold
@velvet fog Thanks for that discussion link. I guess that does point back to my question of what functional aspects of generating the spectres files programmatically (vs manual add that exists now) are roadblocks for any mob flagged as Spectre=Y in db. I assume that it may revolve around Skills that are not existent or have float values for calculating that are hard o glean via query. Or maybe just no one got around to trying it. I'm just looking to determine if there are non-obvious blockers to include dynamic spectre generation from db before I spend a bunch of time trying to code it. ๐
Usually if a seemingly logical and sane approach for a function doesn't exist it's either a) no one got around to it or b) there be Dragons Here.
fathom canopy
Hey guys, I have a couple of questions about usage of the Trade API.
I'm developing a tool to collect and parse data from user postings. The main goal of my project is to gather data to fuel some Analytics and Data Science studies, especially on the Exchange API. In the near future, I'm also considering building a Data Lake to properly store and distribute this information.
Reading the Terms of Use, it seems that my scope conflicts with some points I'm menitoning below:
7. Restrictions: Under no circumstances, without the prior written approval of Grinding Gear Games, may you:
....
a. Adapt, reproduce, store, distribute, print, display, publish or create derivative works from any part of the Website, Materials or Services other than in accordance with the Licence.
...
f. Use any data gathering and extraction tools or software to extract information from the Website or utilize framing techniques to enclose any of the contents of the Website.
...
Main question is: Does these rules apply to data collected from the public API? And if so, is there a proper channel in which I can request permission?
worthy cape
Point f seems to be about the HTML website proper, not the API.
cosmic saffron
email novynn (rory) and he can provide approval
framing techniques probably refers to iframe embeds etc
fathom canopy
I'll send him a mail to talk about this further. Thanks!
And yeah, reading topic F again, it could also be referring to web crawlers, scrappers and such
velvet fog
@buoyant scaffold Added PoB Spectre information in monster page, ex: https://poedb.tw/us/mon.php?n=Redemption+Knight
velvet fog
but no idea how baseFlags created
velvet fog
worthy cape
Reached a milestone, obtained everything before 3.0.0 now.
long fiber
Is the 1 request/s limit both for POSTs and GETs or just the POSTs?
worthy cape
Easiest way to tell empirically is to make one of both and see what the state headers says.
long fiber
True
worthy cape
Rate limits are generally overarching per whole system.
long fiber
Thanks I'll check it out
Ok so it seems they are different.
POST: 'X-Rate-Limit-Ip': '12:4:10,16:12:300'
GET: 'X-Rate-Limit-Ip': '12:6:60,20:12:300'
Does this mean I can do up to 1.5 posts/sec and 1.66 gets/sec? Any idea if these are separate counters or they share it? Also I recall reading @compact isle saying it's 1/s but maybe that's stale information? Lastly, do you know how come the second group of requests:time:penalty have longer penalties but weaker restrictions (say for the POST, 3 req/sec = 10 sec penalty, 1.5 req/sec = 300 sec penalty)?
FWIW I didn't pass my POESSID on this one so maybe it's different if logged in
worthy cape
The limits may change dynamically so whatโs on the response is what holds.
long fiber
Oh good to know they can change around, is that like any minute it can change due to current usage/bandwidth?
worthy cape
The left limit allows bursting while the right governs sustained use.
Itโs more of an explicit change from upstream side, rather seldom but be prepared to handle it and donโt hardcode stuff.
long fiber
Awesome
I get that it's old news but it might be interesting to edit that in your pinned message (I've found the pinned messages board super useful overall)
worthy cape
You can always tell your current state from the โ-stateโ header
long fiber
Ah makes sense I wondered what that was
worthy cape
I think the third number in the tuple changes to the cooldown in case you trigger a limiting event.
long fiber
So it seems the gets and posts are on different counters
I'll try it out lyk
lmao
my pi isn't fast enough
1:6:0,1:12:0
2:6:0,2:12:0
3:6:0,3:12:0
4:6:0,4:12:0
5:6:0,5:12:0
6:6:0,6:12:0
7:6:0,7:12:0
8:6:0,8:12:0
9:6:0,9:12:0
9:6:0,10:12:0
10:6:0,11:12:0
9:6:0,12:12:0
10:6:0,13:12:0
9:6:0,14:12:0
10:6:0,15:12:0
11:6:0,16:12:0
worthy cape
For things like the public-stash-tabs endpoint you have x-rate-limit-policy: public-stash-tabs-request-limit for the group it governs, btw.
Not sure if it's present on all endpoints.
@long fiber
< x-rate-limit-policy: trade-fetch-request-limit
< x-rate-limit-rules: Ip
< x-rate-limit-ip: 12:4:10,16:12:300
< x-rate-limit-ip-state: 10:4:0,22:12:298
Not sure what you mean by that, but as long as all my requests get an x-rate-limit-ip and -state it should be easy to figure out
If you wonder i'm pretty sure gets and posts have different counters, here's the results of a for loop doing both:
POST: 1:6:0,1:12:0
GET: 1:4:0,1:12:0
POST: 2:6:0,2:12:0
GET: 2:4:0,2:12:0
POST: 3:6:0,3:12:0
GET: 3:4:0,3:12:0
POST: 4:6:0,4:12:0
GET: 4:4:0,4:12:0
worthy cape
Endpoints that share a x-rate-limit-policy share their counts.
The search result fetch endpoint has policy trade-fetch-request-limit, while the POST to the trade search endpoint has trade-search-request-limit.
long fiber
Ah ok cool
Dully noted
Thanks for confirming the state penalty decreases I couldnt get to break the rule lol
Welp more methods to write
I assume you have to go above the limit to trigger the penalty? i.e. reaching 17:12 to take your example triggers it, reaching 16:12 is fine?
worthy cape
< x-rate-limit-ip-state: 9:4:0,17:12:0
< x-rate-limit-ip-state: 7:4:0,18:12:299
Not sure which one of those were success and failures, didn't grep for that.
long fiber
i'd assume the first one is a success and the other one triggered it
either way it seems being on the limit is gucci
To pass down the POESSID it's done thru the requests.post's headers and not cookie right?
worthy cape
It's a cookie, which is submitted through the headers.
So yeah, if you've got a jar like that, jam it in there.
long fiber
Ah ok, I figured it would be in headers since I see it in the req.headers' output and the file from last time i've done this is long gone
ty โค๏ธ
Now that you mention it tho, is the 'from' important? I was just passing my user-agent
Ah ok cool
long fiber
Aren't you supposed to be able to requests.get() more than 10 items when you are logged in? As per above with cookies defined as {"POESSID": "abc123..."}. I am seemingly still limited to 10 at a time after which i get a status 400. I recall being able to increase my limit by passing the poessid in but maybe I'm mistaken.
rain jacinth
Hey guys - what do I need to do to pull data from my stashes? Do I need to scrape the WWW or is there a saner way of doing this?
ebon oasis
if i check whats called on the website, you get a json of your stash from https://www.pathofexile.com/character-window/get-stash-items?accountName=Esge&realm=pc&league=Harvest&tabs=1&tabIndex=0
(probably gotta add authorization somewhere in there)
rain jacinth
awesome - thanks for pointing me in the right direction
fathom canopy
Is it possible to "timestamp" the river somehow? Can I actually figure out at which point in time a set of changes happened in a specific change-id if I'm looking at historic data?
Also, does new data comes in a specific timeframe (new change ID every 10 seconds, for example)? Is it possible to catch up "from beginning to right now" respecting x-rate-limits and using a single IP?
I noticed https://poe.ninja/stats shows the latest ID, which is really helpful. I wanted to know from which ID I should start querying if I wanted to look for "changes from the beginning of Harvest league", for example
worthy cape
@fathom canopy You can technically catch up, depending on how rapidly things come in.
Note that old changes may be elided as they've been superseded by future updates to those tabs, so there's less data in the full river now than if you started downloading back then.
As for approximate start-of-league IDs, people sometimes post them on Discord, not sure if anyone did this league.
There's no timestamping, all you can know is that each shard has an increasing ID.
You can't really futz around much with the last-change-id either to try to advance only one shard either, just consume it in sequence treating it as an opaque ID.
mortal bone
You should be able to see your headers in the network tab of your browser
worthy cape
More of an end-user QoL thing, I guess.
"Go make a cup of coffee, you're in the corner for five minutes because some people can't behave."
mortal bone
Yeah, I just meant for the time being
quick basin
any way to get list of atlas maps as raw data?
worthy cape
Most likely, considering that poedb has lists of tiers for each.
@quick basin You probably want to look at AtlasNode.dat where there's fields like AtlasNodeKeys0 and Tier0.
worthy cape
quick basin
I was thinking categorizing the maps into some groups for types of content, mostly based on layout and boss difficulty
i found this on the wiki but im not sure if that is up to date - https://pathofexile.gamepedia.com/Map
worthy cape
I'm not sure whether ninja has everything or just things that have been seen in the river.
simple ravine
Layout and difficulty of them etc is manually curated
quick basin
yeah ofc
just not sure if that is still relevant?
theres a lot of maps to go through lol
simple ravine
quick basin
ah missed that, sorry too tired from playing all day lol
worthy cape
@simple ravine In other news, my project is at 3.5.x now, I really wish there was patch support ๐ฆ
simple ravine
According to version history, there haven't been any significant changes since 3.9
@worthy cape patch support in terms of the ggpk?
i think standalone client patches the content by fetching individual files
worthy cape
Yeah, each download is a full fresh download currently, which sucks balls.
simple ravine
this is why people prefer standalone
worthy cape
I've been considering writing my own tool, but there's a lot of protobuf shenanigans involved.
The upstream tool is in .NET too, so unreadable ๐
simple ravine
which upstream tool are you referring to?
worthy cape
The Steam downloader thing.
quick basin
what are you trying to achieve btw @worthy cape?
worthy cape
To get old releases, standalone patcher assumedly only serves the current-ish release.
@quick basin File format archaeology.
simple ravine
oh, so steam has all the old things
i would not have thought they kept all the old stuff
quick basin
do you guys feel like poe's resources are too scattered?
worthy cape
Looking forward to seeing where new subformats were introduced and for what purpose.
simple ravine
resources? heh, that's a diplomatic way of labelling it
quick basin
theres wiki, poe ninja, moddb, 2-3 client tools for price checking, all kinds of spreadsheets
i dont mind segregation at all, but they seem to be not very connected
simple ravine
because they are made by different people
worthy cape
Different itches to scratch, there's some websites that sometimes succeed in indexing it all.
simple ravine
there have been some efforts to collect stuff like this
i can't remember the site(s), but there are some that portals to different poe resources
I think the subreddit has some info in the sidebar?
quick basin
good idea, didnt check that
simple ravine
the wiki has some too
when I started playing, I spent a lot of time reading the wiki
quick basin
like id be making a spreadsheet similar to this one on the wiki, but id have stuff like is the bossfight in phases or not
thats already unnecessary duplication
worthy cape
The wiki could use a bit of fleshing out again around the maps and the associated bosses, it's a bit of a chore to even find the boss now, let alone how they fight.
Used to be better, I vaguely remember.
quick basin
heh
i mean wiki is info, more like raw data. I understand it should not have player's perspective on best practices and stuff
but wish things were more centralized
simple ravine
yeah, layout grading etc is borderline imo
quick basin
it is
simple ravine
i mean, i like that it's there, but to some degree it's biased
worthy cape
Some things like whether the arena is separate and if it's phased and all that, that's universal information.
quick basin
yeah
i was also thinking, describe map layout linear/free open/closed tight/narrow
then depending on that you could favor legion/breach or not
youd never run legion on a tight map
simple ravine
i guess it kinda depends on playstyle a little too
like poet's pen friendly (is that a thing still?) etc
simple ravine
I mean, hard knowledge ๐ *taps head*
sometimes i feel we create too many spreadsheets for PoE
I mean, they're good to have, but playing the game and learning the maps is probably more fun than sitting in Excel
quick basin
well both is good imho
its easier to learn blocks
and opening a spreadsheet dedicated to harvest/betrayal/divcards helps chunk that
otherwise its a big blob of things you dont know
i do prefer learning over just doing whats in cheatsheets blindly ofc
simple ravine
but yeah, PoE is definitely becoming more and more complex, and easier to play mechanically
worthy cape
Ah, pypoe_ui finally finished parsing...
simple ravine
time to make lunch/dinner
I feel my lifestyle is slipping when I'm home alone ๐
worthy cape
You're chatting with the choir, I'm sleeping twice a day now ๐
simple ravine
I fell asleep in the chair for a while, I like how these gaming chairs are nap-friendly
I've noticed the past year or so, I get so sleepy after I have had food too
cosmic saffron
wfh naps are so bad for your posture
simple ravine
gettig old
@cosmic saffron
mine reclines like this ๐
actually even more, probably
cosmic saffron
mine already has a bunch of pilfered cushions from the couch, it slouches so much
simple ravine
the worst thing though, is if u fall asleep for too long, you wake up with too much blood in your feet
worthy cape
Slightly too short for me, so not super great
violet path
all i do is poe spreadsheets oh no im part of the problem
worthy cape
If they didn't proliferate so much they wouldn't be called spread-sheets.
simple ravine
Hahaha
Caramelized bacon and cheese omelette at midnight. Perfect recipe for heart problems, but I cant resist
pseudo ocean
I don't know if it's the place to Ask this but we are working on a project that is Server defining, related to PoE. If somebody knows or is sort of an "emote maker", contact me, Around 40 emotes for free, in exchange of advertisement and permanent server credit "(insert emote) : made by (insert username) for each emote pinned in a popular Channel. Multiple "Emote makers" are welcome, pngs of in-game arts will be provided. Have a good day. Advertised by a huge, really HUGE bot we are creating.
long fiber
Hi, I've been playing around quite a bit with the post get requests but now I just realized my post requests yield only 100 results and I have no clue on how I can move along the complexity levels with my post requests. Could anyone clue me in?
rapid pagoda
@long fiber What specific APIs are you interacting with, and what do you mean by "move along the complexity levels"?
long fiber
Basically to get the next 100 results on the items API
simple ravine
look for a continuation token
without knowing which API you're using, that's normally how you get the next items nowadays