tooldev-general

[worthy cape]

So the question is for the search trade endpoint that gives you a JSON with 100 items and a total of more, how to access items 101 onward?

I don't know if there is such functionality, the infinitely scrolling page stops at those 100.

@simple ravine It gets an initial JSON of 100 object identifiers and when you scroll paginates in those actual items few-by-few.

[simple ravine]

so you can never get more than 100 items then?

[worthy cape]

I believe that if you desire more relevant items among those 100, you should make more specific questions.

Don't cite me on that tho 😄

[simple ravine]

makes sense

[worthy cape]

(actual trade site renders result as Showing 100 results (590 matched))

[simple ravine]

or index things yourself

[rapid pagoda]

Or sort the results by some property and "paginate" that by changing criteria for that property

[long fiber]

Ok thanks, I figured it would be possible to have more than 100, will have to use some pricing brackets I guess!

[simple ravine]

on another note, I played around in Houdini past couple of days, im fairly happy with the result

off-topic warning, but what-the-heck

[worthy cape]

Those skulls are not raging enough -

[simple ravine]

they dont seem that raging though

[rapid pagoda]

Mildly Peeved Skulls

[simple ravine]

How about a lookup table of sorts

[simple ravine]

I would try to break it down

like, Attack, Spell, Other (for an example)

I am trying to grasp what you want to do with these rules

Ok, I am still a bit confused. The UX does not clearly tell me what this is for 🙂

Ok, how does the weighted sum search work?

Well, I guess that's out of scope here

What is the underlying reason for using this tool, so to say

ah so it simulates using headless (or something) to find what's the nicest thing for your particular build

neat

how do you deal with combination of mods being better etc? or is that out of scope?

ok

so assuming you have a modifier, let's say "+%X to Global critical strike chance"

I see you have a map from pseudo to actual mods (or vice versa)

these are non-pseudo mods, yes?

then you want to combine a set of modifiers into pseudo-mods to keep the search list simpler?

yeah I guess the indexer they made pre-calculates the pseudo-mods, otherwise it would become too costly

I assume also that there are a bunch of ifs and buts, considering this is PoE...

Like, Cold Damage => Elemental Damage is perhaps not applicable each of the times?

I havent played since Legion so I am a bit rusty

exposure attack, is that something new?

so I assume you're brute-forcing values in pob?

considering all the ifs and buts would send you into alice-in-wonderland-mode, taking a more technical approach (like it looks like you're doing) by just taking values and evaluate them

I mean, there are plenty of combinations you would need I guess to really determine in which scenarios you can replace actual mods with pseudo-mods, for things like cold => elemental etc

I normally prefer an object oriented approach, it helps with readability and maintainability. Having said that, what can be interesting is to encapsulate what the pseudo-mod actually "is"

and use that logic to also determine if the list of mods should be replaced or not with this pseudo mod

question: Given an actual modifier, can it be part of more than 1 pseudo mod? I assume so?

So if you could "express" each pseudo mod in a couple of lines of code, and run through each pseudo mod

that could work

it would be a bit more expressive at least

yeah the outcome is probably the same

what I would do however, is try to encapsulate logic in a domain fashion

and then run through them all

sure, let me think about it for a second

i dont know python well enough to write it, can just read it, so I'll pseudo-code it

if I wrote it in C# would you be able to read it proficiently?

I'm gonna think about this for a bit

I dont want to give bad advice

what do you have in the input parameter selections?

and base?

I think Spellslinger came after I took an extended hiatus

hah ouch

I think I recall something similar when making a parser for the translations

[simple ravine]

Ok, took a break etc

So, I assume you would be able to tag pseudo mods with 'Attack', 'Cast' or 'Both'

if modstr["#% increased Burning Damage"] and (modstr["#% increased Burning Damage"] == modstr["#% increased Fire Damage"] == modstr["#% increased Elemental Damage"]):

can you explain this a little

in modstr you have calculated DPS values for each of these mods individually, correct?

so let's say...

    if modstr["#% increased Burning Damage"] and (modstr["#% increased Burning Damage"] == modstr["#% increased Fire Damage"] == modstr["#% increased Elemental Damage"]):
        # Assign the value to our pseudomod
        ret["pseudo.pseudo_increased_burning_damage"] = round(modstr["#% increased Burning Damage"], 2)
        # zero out the mods being used by pseudomod.  Don't delete from list so that we don't crash if checked later
        modstr["#% increased Burning Damage"] = modstr["#% increased Fire Damage"] = modstr["#% increased Elemental Damage"] = 0
        # Add mod to reverse lookup in case mod gets trimmed
        reverse["pseudo.pseudo_increased_burning_damage"] = "#% increased Burning Damage"

#....

    if modstr["#% increased Fire Damage"] and {"Spell"}.issubset(selections) and (modstr["#% increased Fire Damage"] == modstr["#% increased Elemental Damage"]):
        # Assign the value to our pseudomod
        ret["pseudo.pseudo_increased_fire_spell_damage"] = round(modstr["#% increased Fire Damage"], 2)
        # zero out the mods being used by pseudomod.  Don't delete from list so that we don't crash if checked later
        modstr["#% increased Fire Damage"] = modstr["#% increased Elemental Damage"] = 0
        # Add mod to reverse lookup in case mod gets trimmed
        reverse["pseudo.pseudo_increased_fire_spell_damage"] = "#% increased Fire Spell Damage"

Is it always preferred to use burning damage before fire damage if they are the same?

will that include fire damage stuff as well?

so burning damage is a broader term, that will also include all what pseudo.pseudo_increased_fire_spell_damage includes as well

ok, also includes #% increased Elemental Damage in pseudo.pseudo_increased_burning_damage ?

ok cool

that changes my thought process a bit

oh very interesting

if you would have a JSON comprising of pseudo mods and what they include, that could possibly help to create a more generic logic

based on overlap and specificity

but if that logic would be nicer to deal with than your generated code, not sure

do you have unit tests in place?

well, if unit tests break... heh

I was thinking more of encapsulate the unit testing into smaller portions

Like, given mods X, Y and Z, pseudo mods A, B, C etc should be returned

that would help you more rapidly test new approaches, if you encapsulate the logic in a good way

i.e. abstractions and single responsibility principle etc

this takes me back to the original thought a little, which was object oriented encapsulation, which would help you structure what a "unit" in your case is

[rough wren]

I saw that mecurytrade was no longer in development, what tool do people use now?

ty

[mortal bone]

We should just rename this channel something like #..............................tool-development so it doesn't show up on the sidebar haha

[obtuse citrus]

some servers use bots where you have to click on a reaction that gives you access

[simple ravine]

considering the amount of tool developers we have here, im sure someone has the will, time and energy 😄

[hazy fog]

meh

[worthy cape]

@simple ravine Found a small performance problem in my layered I/O code.
On top of a random access file I've got a stream adaptor with a small 8 KiB buffer and a file cursor, to facilitate sequential reads of small primitive types like file headers. This stream adaptor was also used to read the actual file payloads and while I had intended to circumvent the buffer when reading more than 8 KiB, I had not.
There was logic to repeatedly refill that dinky little buffer with small native reads constantly.

Every 8 KiB read had the overhead of creating an event, reading the data, waiting on the event, and destroying it.

[simple ravine]

file headers inside the ggpk file system? like dats etc?

[worthy cape]

Yeah.

[obtuse citrus]

some fun facts... in my dat reader I had to skip using python's .read() and instead only read all the bytes into memory at once because it was completly destroying the performance

[simple ravine]

python isn't really known for its performance 😐

[worthy cape]

I don't need Python to get bad performance 😛

[simple ravine]

heh

[indigo relic]

anyone has an array with all the map mods ? dont want to have to convert them from https://pathofexile.gamepedia.com/List_of_map_mods 🙂

[golden bane]

@indigo relic You can get them from the wiki API: https://pathofexile.gamepedia.com/Path_of_Exile_Wiki:Data_query_API

[indigo relic]

thx 4 the tipüp

[worthy cape]

How would one sift it out of RePoE mods.json, grep for "domain": "area"?

[indigo relic]

as im new to the api i hav still no clue on how to query for the map mods ... any way i can look at some more example biside the once on poe ?

[worthy cape]

You can see the queries the tables are using by using the "View source" tab of the page. Should be "straightforward" to translate that into a cargo query.

[simple ravine]

https://github.com/brather1ng/RePoE/tree/master/RePoE/data @indigo relic

the throwback to youth when reading David Eddings btw

[indigo relic]

😉

[rain jacinth]

guys quick question - is there a tool that lets me monitor prices of the items I have on sale?

[broken cloud]

@indigo relic If you still need it, I have a manually maintained list of area mods (not guaranteed to be complete) https://github.com/briansd9/exile-diary/blob/a8bc42b327e6137416cdb95829e5f36b55b3b0db/res/data/constants.json#L2419

[indigo relic]

thx

[rain jacinth]

If I want to pull characters and stashes do I need to alleviate through CORS proxy or is there a nicer way to go about it? I got it working with a proxy but obviously don't like it.

[worthy cape]

I'm starting to regret picking xdelta3 for my delta patching needs. The C project's Windows support seems sketchy and the project files are rather rotten, while the popular Rust crate only exposes two functions:

pub fn decode(input: &[u8], src: &[u8]) -> Option<Vec<u8>>
pub fn encode(input: &[u8], src: &[u8]) -> Option<Vec<u8>>

That decode doesn't exactly scale well for 30+ gigabyte output files 😄

[worthy cape]

Bonus points - the crate doesn't even work.

[worthy cape]

Clowns explicitly coerce the size of the input to uint32_t, despite the signature taking a full size_t. Amazing.

[simple ravine]

what's xdelta3?

[worthy cape]

An implementation of VCDIFF, a format for delta patches.

From old.ggpk and new.ggpk you may generate old-to-new.xdelta3, where later given old.ggpk and the delta you can regenerate new.ggpk.

[simple ravine]

ah I see

[worthy cape]

The delta patch contains instructions on what data to sample from the original file along a sliding window and inlines data that isn't shared.

[simple ravine]

hmm, i just had a hilarious idea

blockchain

[worthy cape]

...

[simple ravine]

isn't that pretty much diffing, kinda

[worthy cape]

Not quite. Blockchain techniques brute-forces solutions to a signing problem where you have an ancestry and some payload, creating trust in that you cannot go back and change history as the chain will have carried on afterward.

[simple ravine]

well the bruteforce is an implementation detail

not all blockchains use this, if i understand it correctly

like RNDR token

there is this one for storage as well

[worthy cape]

The point is that there's a proof of significant work, such that it cannot be spuriously fudged.

[simple ravine]

either work or stake, right

[worthy cape]

Core to most techniques is that the value is in the payload you can prevent tampering of.

[simple ravine]

but yeah, you are right, not very suitable

[worthy cape]

As for fancypants contract tech, I don't know or care, I've got all relevant terms blocked on Twitter since forever.

[simple ravine]

hehe

[worthy cape]

The whole field is like ten nines of scam.

[simple ravine]

yeah, that's why I lost my interest as well

99% opportunists

[worthy cape]

In any way, I was making a small helper tool to rehydrate a series of GGPKs from deltas and ran into this fun behaviour.

[simple ravine]

size_t same as word/dword?

i haven't touched c/c++ since high school

and that was a long time ago

[worthy cape]

size_t is the integer type that can hold the size of an object, including arrays.

As such, it's also suitable for indexing arrays.

[simple ravine]

Clowns explicitly coerce the size of the input to uint32_t, despite the signature taking a full size_t. Amazing.

I was just trying to understand this.

[worthy cape]

The native function is like xd3_decode_memory(void const* input_data, size_t input_size, ...).

The Rust bindings call it as xd3_decode_memory(input.as_ptr(), input.len() as u32, ...), intentionally ruining it for 64-bit platforms where arrays can be larger than an u32.

Oh gods, even worse, they blindly estimate the maximum size that the output will be, something which I don't believe universally holds.

This library goes in the bin.

[simple ravine]

so size_t is platform-dependent, like word and dword?

[worthy cape]

Aye.

Pretty much all types in C are, apart from char, uintNN_t and their friends.

[simple ravine]

well, aren't all int 32 bits?

[worthy cape]

Hell no.

[simple ravine]

long being 64, no?

[worthy cape]

No.

You're tainted by .NET 😄

[simple ravine]

😄

my life is easier

[worthy cape]

This table should scare you - https://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_models

You've got some types of varying sizes in .NET, you may have been exposed to IntPtr

It's analogous to C intptr_t, suitable for holding either an integer or a pointer cast to an integer.

One of the most common portability problems with 64-bit C code to Windows is that the hippies hard-assume (either out of ignorance or malice) that long is 64-bit.

[simple ravine]

https://stackoverflow.com/a/18971763/1447764

they must have had a lot of acid and crack cocaine when coming up with this shit

[worthy cape]

Historical reasons.

Most design decisions made sense at the time, and many are made with the future in consideration.

Every time a core language tries to make behaviour stricter or constrain sizes, there's always some angry luddites being upset.

[simple ravine]

what i meant was "long long" "long int" "int"

like what

stahp

"long long int" 😂 🔫

[worthy cape]

@simple ravine Alternative spellings of the same type, due to how the language evolved from prior art and consolidates multiple dialects into a standard.

The C standard is a huge hecking accomplishment.

Why it's still the way it is? You're not going to earn any points breaking existing code for no real benefit.

[simple ravine]

I don't get the difference between long long and long long int

[worthy cape]

They're identical, apart from the spelling.

[simple ravine]

This is why I never going back to c

screw that

[worthy cape]

You're talking about a language that didn't have argument types back in the day, and where you could arbitrarily use struct members to index into any memory.

@simple ravine No-one's forcing you to use C, but as long as it's the lingua franca of interop, it helps being aware of how it is.

[simple ravine]

For sure

what bothers me is the "at least" part

[obtuse citrus]

there there is also this :P

https://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_models

[simple ravine]

I want to know exactly the size of my structs

so I can optimize for them

[simple ravine]

yes, zao linked it a few minutes ago omega 🙂

[worthy cape]

@simple ravine That's why we have sized types.

With good names, and not the int, long, etc. bullshit that .NET insists on.

[obtuse citrus]

oh derp

[simple ravine]

well, as long as you're aware of the size, naming is just a convention

[worthy cape]

I'd love for C# code to prefer Int32 and UInt64, but that ship has sailed.

[simple ravine]

and it's convenient that they don't change

[worthy cape]

(those are CTS types)

[simple ravine]

I mean, you can use that naming in C# without any drawbacks

[worthy cape]

I like that it's harder and harder to reason about the layout of types in C++ now, found out about [[no_unique_address]] the other day.

That attribute helps the compiler give fewer fucks about things like "each subobject should have an unique address" and let you get empty-base like behaviour for your data members.

[simple ravine]

You lost me there

[worthy cape]

Every member in a C++ class traditionally needs to occupy at least one byte.

This may unnecessarily bloat the size of your classes if you have several such members.

They've finally formalized a way to waive that requirement.

[simple ravine]

I'm not sure I get the use case

[worthy cape]

It's not uncommon to have objects with no data but a bunch of functions. If you want to hold one of those are a member of your type, you're paying for it despite it holding no data.

[simple ravine]

ah yes

[worthy cape]

The attribute lets you have those as a member where they'll overlap with other parts of your object, but it's fine as there's nothing underneath to access anyway.

[simple ravine]

I've used that quite a bit with generic to optimize i c#

[worthy cape]

Anyway, I'm now trying to figure out how to get CMake to generate proper definitions for this library.

CMake - the least horrible build generator tool.

[grizzled seal]

@fickle yew just a heads up there's a couple different errors on the poe.ninja data dumps page (metamorph label duplicated once, blight start/end date duplicated twice)

[fickle yew]

@grizzled seal Thanks

[worthy cape]

3.10.* obtained, there's an end in sight.

Got to get coding on the actual tooling for outlining GGPKs now 😄

[spring hemlock]

Hi All! I want to use POE oAuth for my "yet another" trading helper. But I can't found any public information about it 😦 please advice me.

[worthy cape]

@spring hemlock There's an email address in the pins you can contact.

[spring hemlock]

Thank you!

[pseudo ocean]

https://github.com/Caseyy89/FlippingCalc/releases/tag/2.0.3 finished adding fragments in, scarabs will be next, also condensed it down to just 1 link window

[simple ravine]

@pseudo ocean Your flipping thing

Will you support (or does it?) flipping alts to chaos and back?

sometimes, you want a 3-way arbitrage when flipping

[viscid cairn]

guys i need updated program toi calculate chaos per hour any 1 have bcs currency cop dont show seeds

[simple ravine]

Exilence?

[pseudo ocean]

@simple ravine how do you mean ? You can do alts for chaos

[simple ravine]

Ok, I guess the UI is confusing me then 🙂

[pseudo ocean]

What's confusing you ?

[simple ravine]

Ah, so the values are kinda arbitrary to whatever u want

[pseudo ocean]

Yeah , the links are pointing to chaos/x and x/chaos tho

[simple ravine]

Right, when I did flipping I would perhaps to 3-way arbitrage to maximize profits

or something like that

[cosmic saffron]

the arbitrage calculator's called strongbox or something

[simple ravine]

so let's say Buy alts for chaos, sell chaos for fuse and then fuse to alts

[pseudo ocean]

Oh

[simple ravine]

https://www.investopedia.com/terms/t/triangulararbitrage.asp

[pseudo ocean]

Yeah that would add A LOT more windows there is so many different combinations

[simple ravine]

well I guess it depends on how you build up the UI

maybe I'll take a crack at it

[pseudo ocean]

Basically, I suck, I started it to learn c#, it's getting more advanced as I look into more things I guess

Feel free to fork, it's a mess tho ;D

[simple ravine]

yeah I saw 🙂

I was like "Where's the code?"

"Oh, it's all in this one file"

[pseudo ocean]

Kek

[simple ravine]

I guess the use-case for my thing is a bit more for advanced users, though

different audiences

pro tip tho: read a little about object oriented programming and SOLID principles

perhaps the basics of Aggregates and DDD

helps you structure things a little

[pseudo ocean]

I know these words individually but not in this context :D

[simple ravine]

oh well

Taking some inspiration from node-based applications like Houdini, one could potentially make something like this

[pseudo ocean]

Def. Not something I could do

[worthy cape]

@simple ravine I preach main()-oriented programming 😛

[simple ravine]

just have to break it down into small things

@worthy cape that's because your perspective is mainly from an automation perspective, i.e. sysadmin 🙂

no pun intended

something like that perhaps

and then simulate

would be badass if it read the player's public stash info and filled things in automatically too

[mortal bone]

We did this for a graphing class one time. We graphed out currency exchange rates and found loops in the graph

[pseudo ocean]

thats too 5head for my skills, c# is my first and only language and im still learning :S

[worthy cape]

Has anyone ever seen a GGPK where a single FILE or PDIR is referenced from multiple PDIR lists, or is it always 1:1?

[simple ravine]

I have never seen that myself

actually, I am not sure if I would have noticed, now that I think about it for a bit

[worthy cape]

@simple ravine Download completed btw, 53.3 GiB patches and 343 GiB of "latest GGPK each league"

[simple ravine]

You have all the data!

[worthy cape]

Haven't checked if it's all intact yet, and still have to grab the tags for the first half.

May or may not have broken the validation tool recently.

[simple ravine]

oof

[worthy cape]

Prototyped it with entangling between the indexing and scanning phases, now it's factored out so I can have the same robust logic for both validation and regular VFS use.

Right now I'm writing the tool to "outline" a GGPK into a digest-addressed directory tree and a skeleton GGPK.

[simple ravine]

hmm

i just had an abstract thought

[worthy cape]

Could probably even hack PyPoE to understand this format, if it can be signaled somehow that it's dehydrated.

[simple ravine]

with handwaving gestures
something-something reverse index á la google, but with SHAs

ae43c19ea911534e => ggpkdata/3.10/Data/Foo.dat

and/or symlink

[worthy cape]

For what purpose, tracking renames and what packs a file originated in?

You could explode a GGPK into a tree with symlinks into the hash-tree, sure. Would eat inodes like crazy.

[simple ravine]

hmm true

well I was thinking like instead of having these diffs

you store each unique file variant

would be a simpler, probably less storage-efficient

but easier on compute and brain wrangling

[worthy cape]

@simple ravine I've got two things I want to have.

1. a full set of pristine GGPKs (or the ability to reconstruct them bitwise identical);
2. skeleton trees of all GGPKs pointing to all the file contents deduplicated.

Case 1) I only really need in case I screw something up or figure out something I want to do with them, like look at the content of FREE blocks, or want to pivot to some other storage type for 2).

[frigid nova]

Does anyone know if the instructions which show up on items and in item clipboard entries show up in the ggpk?

e.g. All sockets must be filled with Fossils before this item can be used. Shift click to unstack.

[worthy cape]

@frigid nova ```
Running target\release\examples\binary_search.exe 'C:/Games/Path of Exile (Standalone)/Content.ggpk' 'All sockets must be filled with Fossils before this item can be used.'
33433868965_1593864627883431000
Data/ClientStrings.dat: [788174]
Data/ClientStrings.dat64: [874126]

(tool searches for UTF-16LE text in all files in a GGPK)

Huh, that's odd.

Id ItemDisplayDelveCurrencyNotFullDescription in Data/ClientStrings.dat holds the first sentence of your string.

Id ItemDisplayStackDescription holds the second sentence.

[worthy cape]

Accidentally the whole thing

[simple ravine]

hah

[obtuse citrus]

they are linked from currencyitems

[velvet fog]

I can't find any link from CurrencyItems.dat to ClientStrings.dat

[rapid pagoda]

I wouldn't be surprised if those messages are generated server-side, so there's no connection visible in the client data

[velvet fog]

just search all .dat files, can't find any reference to ClientStrings.dat

[frigid nova]

Ah okay, thanks for looking.

[obtuse citrus]

Ok I misrenembered

the link is from the .ot file the item inherits from

that goes for the grey help messages on items

Base -> description_text

field specifically

see also the wiki export https://github.com/OmegaK2/PyPoE/blob/dev/PyPoE/cli/exporter/wiki/parsers/item.py#L2636

Also some of clientstrings is linked to by the mod descriptions as well

reminderstring <clientstrings.dat id>

[velvet fog]

but for Prime Chaotic Resonator, can't find any description_text about ItemDisplayDelveCurrencyNotFullDescription or ItemDisplayStackDescription in Metadata/Items/Delve/DelveStackableSocketableCurrency.ot

[lavish gust]

Is there any list of DropLevel propety values for different items? My purpose are item filters, searching for information about this property

[velvet fog]

there is a ItemLevel column in BaseItemTypes.dat

[lavish gust]

this is something different

[velvet fog]

that's only thing we have

[hazy fiber]

afaik its only certain uniques like skyforth and voidbattery that have a drop level != required level

if theres one for currency I have no clue

[lavish gust]

I'm searching for the value of this propety for all kinds of items (currently at worst I can just filter test them). My discoveries so far: Offering is drop level 1, Divine Vessel - 68

[civic hawk]

Hi! I got a question about poe.ninja API.

I wanna grab an Exalted Orb buy-out price and put it straight into Google Sheets. Any idea on how to make the XPath actually work?

[simple ravine]

@civic hawk Helps if you're describing what you are trying, and what you have done so far. Google Spreadsheet is a little niche

[civic hawk]

just trying to grab the exalt price into the sheet

thats it

i just dont wanna update it myself

[simple ravine]

figured as much

[civic hawk]

the rest of the sheet operates on this number

so i need it updated all the time

well theres really nothing more to it

i just need the number to be there and thats it

[simple ravine]

i just googled ur problem and found a lot of resources

so not sure what's the issue

[civic hawk]

the issue is that whenever i copy the xpath of the actual number i get an error

1 sec gonna reproduce it once more

[simple ravine]

you using ImportJSON?

[civic hawk]

so thats it

i didnt know you need to import json into it found some script to do that so i should be fine now

anyway ty

[simple ravine]

I didnt know about it until a minute ago when I googled ur problem lol

I guess I am a glorified googler-assistant

[civic hawk]

well i googled that problem too but couldnt find anything proably bcs of poe ninja in the search term

[simple ravine]

I just googled xpath json google sheets

[civic hawk]

i actually did too

didnt get anything like this tho

only json is missing

anyway it works now

so yeah

[simple ravine]

🎉

[violet path]

@civic hawk pm'd about sheets
https://github.com/bradjasper/ImportJSON and https://github.com/qeet/IMPORTJSONAPI are both good options for making an importjson formula since google doesn't have one by default

[long fiber]

Is anyone aware of pathofexile.com/trade chaos equivalent ratios? They don't seem to match poe.ninja and it is otherwise a hassle to extract while it's maybe already done. Thanks!

[cosmic saffron]

there are trade helper browser extensions which shows equiv in mixed ex + chaos but those won't change the sorting order

[violet path]

I think you're going to have a hard time finding out very precise, up to the minute ratios because it's also very difficult for people; have to somehow know the ppl who are fixing/afk/aren't responding. the exchange page does have an api you can query though, can try filtering the listings yourself

[grave wren]

@velvet fog using ublock in firefox i've disabled it for poedb but it still says that it recognizes an adblocker, any idea why?

[cosmic saffron]

is umatrix still active?

[velvet fog]

press F12 to enter dev mode and see if any script blocked

[grave wren]

hm weird, it blocks some google scripts

ah it's from firefox itself apparently

[velvet fog]

did Firefox have Incognito mode?

[grave wren]

it does but it also has some privacy protection on by default

https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Privacy/Tracking_Protection

this did it

[worthy cape]

@long fiber I believe that the official trade site has a ratio that's updated when it drifts too far from what the "economy" considers currency to be at. The reasoning behind it is that it's quite expensive to do with a ton of database load.

[fickle yew]

@compact isle It looks like there's an issue with the Hardcore Harvest ladder - it is reporting some characters as dead even though they're not - one example: https://i.imgur.com/GPYFLam.png + https://i.imgur.com/TDDavku.png

[worthy cape]

Dead man walking.

[compact isle]

that happens in any hardcore league, nothing new

[simple ravine]

now I am curious as to how that happens

[velvet fog]

delete character after dead and renew same name

[simple ravine]

ah of course

[fickle yew]

@velvet fog No that is detectable through the character ids in the ladder api. This is something else..

[compact isle]

it can happen when someone dies while disconnected and the ladder entry gets saved but the character doesn't

if the player contacts support we can reset their ladder entry flags

[velvet fog]

@compact isle garena online api seems broken, return same result

[compact isle]

yeah I'm working on upgrading their trade site atm

should be updated now

[velvet fog]

works now, thanks

[worthy cape]

I wonder if a linear scan from byte zero through the file gathering FILE/PDIR/FREE chunks would be less wasteful than the random access fandango I do now.

Does anyone know if there’s any holes between chunks with bogus data or if they’re always densely packed? What happens if there’s not enough room for a FREE header when a block is partially reused, is it just not considered a candidate?

[frigid nova]

Anyone know where the cluster jewel enchant mods, and translations, live? Looks to me like theyre currently not in repoe:
Added Small Passive Skills grant: 6% increased effect of Non-Curse Auras from your Skills

[mortal bone]

Those all live in the tree json

they are nodes without groups iirc

[frigid nova]

Presumably they have corresponding mods too, since they are enchants on the item itself?

[mortal bone]

The item contains a list of ids that it grants which are associated with the "skill" id in the tree json. The skill id is just a reference to a node id

[simple ravine]

@worthy cape I can't recall finding garbage when I made my linear reader

[lavish gust]

Anyone knows what's the item class of captured itemized beasts?

@pseudo ocean I'm interested in your project, although I have my own that is not a filter generator but compiler (it has own extended-filter-language that both adds extra syntax and fancy stuff like poe.ninja/price-dependent-items)

[lavish gust]

@pseudo ocean Looks like we have very similar goals (GUI config, poe.ninja data) although by reading your code I can't get how it actually works and what some keywords mean. You seem to have created a separate language while I did syntax extensions. My project is already documented so you can look at the examples, tutorial and tests on https://github.com/Xeverous/filter_spirit. Now I'm just working on the graphical end and filter debugger functionality.

[worthy cape]

Made some nice optimizations over the last few days.
Instead of random-accessing all over the disk a few hundred thousand times I scan the file linearly from the start for possible FILE/PDIR entries and parse those as I do, building the tree over those linear entries instead.
I also moved to memory mapped I/O again which lets me care less about buffered reads. Moving to sqlite3 instead of a file tree saved me a lot of read-back for data integrity checks, if it's in the DB it's complete.
I can slam through a 8 GiB file from a remote SSD now in around 30s, which is kind of fast enough.

[lavish gust]

@pseudo ocean can you do voice or some chat elsewhere? I guess we have a lot of info to exchange

[lavish gust]

@pseudo ocean ok, I get how your code works now, and you get mine too. Do you have any idea for a feature to support filter variants? Also, can you should any one that uses poe.ninja data?

[lavish gust]

You werent searching enough. poe.ninja has own auto-generated documentation: https://poe.ninja/swagger/index.html

[lavish gust]

@velvet fog is there any public API for PoEDB? I'm interested in base item names, mods etc

[lavish gust]

reading JSONs is orders of magnitude easier than simulating the website content

[civic crane]

what's the purpose of these bytes in column which is FK to another .dat

[worthy cape]

It's not two distinct ref fields then?

Maybe the second is a NULL string?

[simple ravine]

32 bit vs 64 bit

[worthy cape]

@simple ravine But aren't FKs 4/8 bytes normally?

[simple ravine]

I can't recall, but because they're storing the integer values as bytes, they store the whole thing

[worthy cape]

(been way too long since I cared about .DAT, going to have a helluva time once I get my multi-GGPK thing going)

[simple ravine]

apparently

so it's easier & faster to read them into in-memory structs, I guess?

[worthy cape]

Not quite sure what you're saying, but DAT fields are always fixed-width to the schema.

@simple ravine In completely other news, I've ingested 1.0.0 through 3.3.2 now, 56.1 GiB of sqlite3.

[simple ravine]

what are you storing in the db?

[worthy cape]

(digest BLOB UNIQUE NOT NULL, payload BLOB NOT NULL)

The file contents, indexed by digest. 757914 unique files thus far.

[simple ravine]

no file path?

in case they move something

[worthy cape]

File location doesn't influence the digest, just the contents.

I also keep a gutted index GGPK that has the same tree as the full GGPK, but all files are truncated to 0 bytes.

[obtuse citrus]

pointers double in size with 64 bit

be it to other files or to self

[worthy cape]

I wouldn't assume that it's part of the FK.

@simple ravine

[obtuse citrus]

64 bit unsigned ints to other files

32 bit unsigned ints to self

[simple ravine]

nice, zao 🙂

[worthy cape]

@simple ravine I've been toying with having a different representation for the trees as well, but haven't come up with anything that would be more aligned with my needs yet.

[civic crane]

@obtuse citrus yep, thats about green part, but what about red part?)

[simple ravine]

perhaps in sqlite as well, zao?

[worthy cape]

I want to do things like "for all SMD files across all versions, do X".

@simple ravine I've got a feeling that it'll be hard to beat the dense trees with something like a parent/child table and an entry table.

[simple ravine]

well, if you have them in sqlite, you have the query engine that will help you do that "rapid development / experimentation"

[worthy cape]

Not rapid if the queries suck mud tho 😛

@simple ravine I've also considered hacking existing tooling like pypoe into understanding these gutted indices + DB, so I can cheaply open packs.

[simple ravine]

well, all you need is 3 columns (or 4)
ggpk version, name, digest (and optionally file extension for fast query)

name would be the full path, or perhaps splitting it up into path and filename

[worthy cape]

Time will tell... been focusing on getting ingestion going as it's a many-day process.

For 3.4.x, it takes around 40-50s to ingest a file that's mostly in the database already, 90s for the first one of a league.

I also need to reconsitute the files up-front, which also takes time. Not enough SSD to keep more than one league at a time.

[worthy cape]

@simple ravine Finally done with DB ingestion of packs, started on Friday morning and just now done.
1171394 total unique digests, 139 GiB database size.

Should probably find a SHA256 extension and actually validate the contents too ^_^

[simple ravine]

thats a little too big for an sqlite database though? not an expert on sqlite, but my gut feeling is squeaking

[worthy cape]

I've had foo_wave_seekbar users with a signature database of hundreds of gigs, was rather reliable.

(16 KiB data per track, 50-100k tracks and up)

[simple ravine]

yeah I guess it depends on access patterns

[worthy cape]

Quite hyped to be at a place where I can get some exploration done. Still got the separate indices tho.

[simple ravine]

Yeah, will be interesting to hear about your findings 🙂

[worthy cape]

sqlite> select quote(digest), quote(digest("sha256", payload)) as computed from files where quote(digest) != computed;

Odd perf... if I copy the database while running a query against it, I'm getting 119 MiB/s on the copy and 120-some megs a second for the query. If I pause the copy the query goes at 500-600 MiB/s.

I thought SSDs would be better at concurrent use.

All data integrity is intact \o/

[simple ravine]

maybe not the disk itself?

[worthy cape]

148,524,338,499 bytes of actual payload in 149,295,296,512 of database, nice overhead 😄

[simple ravine]

that's fair

not many indexes though, I presume

735 MB of index and other things on 1,171,394 entries

[worthy cape]

Mother of fuck. I screwed up my stripped GGPK files 😄

Off-by-one byte on the size of the name...

return 4 + 4 + 4 + 4 + 32 + (dir->name_.size() * 2 + 1) + dir->entries_.size() * 12;

Trying to decide now if it can possibly be repaired in-place, where I would need to traverse all broken files and figure out how much to shift the directory offsets; or if I should sink four additional days into re-emitting them.

[simple ravine]

ooouuchh

[worthy cape]

Upside of things, the bug has been there all along.

End result is that a chunk is shifted by N bytes, where N are the number of PDIR entries preceding it in file order.

[simple ravine]

should be able to reconstruct them with other words, question is if the work to do it is worth it

[worthy cape]

I'd like to borrow a clown nose. Had a second (correctable) bug where I forgot to consider the GGPK and FREE up-front chunks in offsets too.

[agile raptor]

i tried messing a bit with POB lua scripts, to make it work with Tencent accounts, and it partially works. It parses characters list and imports character tree, but cant import items data. I suspect, that because it's in Chinese If someone wants to try it yourself, replace www.pathofexile.com with poe.game.qq.com in ImportTab.lua, PassiveSpec.lua and PassiveTree.lua

[velvet fog]

there is a chinese modified version of pob

[agile raptor]

thanks, gonna look for it 😛

[velvet fog]

https://dev.tencent.com/u/ZdDroid/p/POB2CN/git/releases
but I can't visit right now

[agile raptor]

found it here http://bbs.17173.com/thread-10923378-1-1.html

[lavish gust]

Parsing item metadata json, can someone enlighten me which maps are which and what is the pattern for recent maps?

[velvet fog]

Metadata LIKE 'Metadata/Items/Maps/MapWorlds%'

[lavish gust]

@velvet fog thanks for confirmation, looking at the wiki's data there is an obvious trend:

original
Metadata/Items/Maps/MapTier4_6

awakening
Metadata/Items/Maps/Map2Tier4_6

Atlas of Worlds
Metadata/Items/Maps/MapAtlasPromenade
Metadata/Items/Maps/MapAtlasPromenadeShaped

Synthesis
Metadata/Items/Maps/MapWorldsPromenade

Betrayal
Metadata/Items/Maps/MapWorldsPromenade

Legion
Metadata/Items/Maps/MapWorldsPromenade

Metamorph
Metadata/Items/Maps/MapWorldsPromenade

Blight
Metadata/Items/Maps/MapWorldsPromenade

Delirium
Metadata/Items/Maps/MapWorldsPromenade

Harvest
Metadata/Items/Maps/MapWorldsPromenade

Can I obtain map tiers from this file? There seem to be no obvious connection between drop level and map tier.

[velvet fog]

check MapSeriesTiers.dat

[lavish gust]

thanks, looks like I will need to use more than just metadata

[lavish gust]

WTF is this item?

"Metadata/Items/Weapons/OneHandWeapons/Daggers/EtherealBlade1": {
    "domain": "undefined",
    "drop_level": 1,
    "implicits": [],
    "inventory_height": 3,
    "inventory_width": 1,
    "item_class": "Dagger",
    "name": "Ethereal Blade",
    "properties": {
        "attack_time": 667,
        "critical_strike_chance": 600,
        "physical_damage_max": 8,
        "physical_damage_min": 4,
        "range": 10
    },
    "release_state": "released",
    "requirements": {
        "dexterity": 9,
        "intelligence": 6,
        "level": 1,
        "strength": 0
    },
    "tags": [
        "not_for_sale",
        "dagger",
        "attack_dagger",
        "one_hand_weapon",
        "onehand",
        "weapon",
        "default"
    ],
    "visual_identity": {
        "dds_file": "Art/2DItems/Weapons/OneHandWeapons/Daggers/Dagger1.dds",
        "id": "EtherealBladeHeld_"
    }
}

Anyone seen this? Seemingly released, wiki has no info on it.

[simple ravine]

Quite possibly some junk data in there

[lavish gust]

everything like this had release_state either unreleased or legacy

[simple ravine]

Hence, "junk"

because drop rate can still be 0

As it can be to support stuff like Blade Vortex

[worthy cape]

Maybe it’s the ones that manifest from the Blade skills, for animation/detonation/etc?

[simple ravine]

This spell creates ethereal blades which orbit in an area around you [...]

[rugged night]

What changed on poe.ninja between 15 and 39 hours ago?

Its now sending 25mb of backslashes when the backend (selenium) of my app requests an update... instead of the page it asked for.

[rugged night]

Its like some sort of encoding thing he's using is messing up... and keeps trying to change \ into \\ (recursively)

.qc-cmp-small-toggle.qc-cmp-toggle-on, .qc-cmp-toggle.qc-cmp-toggle-on { background-color: #778890!important; border-color: #778890!important; } #qc-cmp2-ui .qc-cmp2-summary-buttons button[mode=\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

[worthy cape]

There's no escape from these escapes.

@fickle yew Site's hosed? ^

[rugged night]

Those slashes then go on for about 25mb

Its gonna be hammering his usage quota if anyone else is getting daily updates from poe.ninja with an automated backend

[worthy cape]

Do you get similar results with an interactive browser, or is it doing something weird based on user-agent or something?

[rugged night]

Looks fine in normal chrome that has cookies, etc

When using Selenium though... its suddenly started dumping huge amounts of \\\ into the first HTML that loads, when asking for maps page, etc

[worthy cape]

how's curl and friends?

[mortal bone]

Is there a reason why you aren't using the api?

[rugged night]

Nope - just wanted to practice something with Selenium... and I only update prices once a day... so was low enough usage it wouldn't cause an issue 🙂

[mortal bone]

I get scared when I hear "backend Selenium"

[rugged night]

Same reason I dont use a neural network library (for now) and wrote my own NN & matrix math libs... learning how they work

I've stopped the app running while poe.ninja is having this issue though... dont wanna suck his usage quota dry by trying to re-grab a failed page constantly

Other people using the site with their own tools might not have noticed poe.ninja's change in response / behaviour -.-

[grave wren]

i dont think many use selenium when a rest api is available

comparison between a network library and consuming http content is also a bit arbitrary imo

[rugged night]

Its just other things I'm messing with & learning atm

No real reason for choosing / making / using any of them

Yes, there are "better" ways... but sometimes you gotta just other stuff.

I'd never used Selenium before this

I could probably code around poe.ninja's problem... or change to the API... and not bother mentioning it here... but that wouldn't help the dude if other people are hammering his site atm

[worthy cape]

@velvet fog How did you deal with changing schemas over time for your DataHistory page, guesswork and looking at pypoe history, or do you have old schemas around?

I'm pondering whether to implement older schemas or just not bother much with Data/ for anything but current league.

[rugged night]

(I think he reads these)

[grave wren]

dunno site seems responsive either way so i wouldn't worry, i'd still recommend consuming apis when possible and not use selenium if not needed

[rugged night]

Oh, its def. not needed... it was a quick test to see how Selenium worked

Stage 2 of my app, if I can be assed to code it... needs a neural net or something to guess prices... and consume the TradeAPI

Basically just because I wanna learn NN stuff... and see what it can/cant do

I put off learning it for years... thinking it was really hard because of all the math I saw in books, etc...

...but have been quite upset so far at how easy they are to code :/

A perceptron I wrote playing "high / low" without being told what the line (slope) is first: https://youtu.be/pxfxDET0ZQo

I thought it looked pretty... so shared with a couple of friends

[fickle yew]

@worthy cape what? I don't see any issues or reports of issues with the site?

[worthy cape]

@fickle yew The problem @rugged night mentioned above with Selenium as user agent - https://discordapp.com/channels/174993814845521922/175290321695932416/743148603630944267

[rugged night]

The snippet I provided of the HTML is from https://poe.ninja/challenge/maps

Its near the start though... so I'm assuming it'd be returned on any page

[fickle yew]

I'll take a look in a few.

[rugged night]

If you want the full HTML returned... just lemme know and I can shove it in a zip for you

Its like 25mb of backslashes though

...so should be easy to spot

My guess would be some encoder/decoder is messing up

[fickle yew]

Also @rugged night why are you fetching the html when the site has a perfectly fine api?

[rugged night]

Wanted to learn how to use Selenium... and my app only checks once a day, so wasn't gonna be too much of a load

[simple ravine]

heh

[fickle yew]

Please don't. It flags my ads with invalid traffic.

[rugged night]

Ahhh... never thought of ads

I shall switch to API

[fickle yew]

Thanks. I wonder if it's actually adsense that's giving you a bunch of backslashes.

[rugged night]

Looks more like something it trying to encode/decode \ to \\ to me

...and its getting stuck in a loop

Could be either though

[simple ravine]

Genzzry, I recognize your nickname quite a lot

were you the guy who did a bunch of F# stuff here before?

[rugged night]

Nope

I probably did some C# stuff... I made a TradeAPI app that logs all data to my HDD

(so that I only need to grab it once & can play with the data later)

[simple ravine]

Oh wait, you were in the Lifting's discord before right?

[rugged night]

Dont think so

[simple ravine]

hmm

[rugged night]

I'm a mod on Don's Twitch... maybe that?

[fickle yew]

@rugged night if you're only doing one refresh per day it probably doesn't matter. But I think your results will be a lot cleaner using the api.

[simple ravine]

Nah, not that. I recall having a meaningful discussion with you in the past anyways. Oh well..

[rugged night]

Checked my logs... and we chatted back on 04/08/2017 via PM about a league start, helping eachother level, etc

No context about where we met or anything

^ @simple ravine

[simple ravine]

Pretty sure it was from Lifting's Discord, but I might be mistaken.

[rugged night]

I PM'd you a link to an autism video... then we instantly started talking about ggpk download for league start and helping eachtoher level, gear-up, etc...

...so must have met before the PMs

shrugs

[long fiber]

Does anyone know how long an item is findable by get request with search I'd? Or how long until it is delisted

[velvet fog]

@worthy cape DataHistory only read first int for each version

[velvet fog]

find out 1.0.0 spec will consume a lot of time

and most old data is useless

[worthy cape]

Oh right, I forget about the first count.

[worthy cape]

@velvet fog I'm wondering a bit over the tabbed layout of poedb, is it possible to make individual tabs of pages linkable? I'm linking people resources for a Bingo league and it's a bit cumbersome to tell people to go to the base URL and then click a specific subtab.
For example, I would like to link to "Atlas BaseType Drops" of https://poedb.tw/us/Atlas, but the URL doesn't change when you click the tabs and the anchor you get if you try to copy link of the tab either does nothing or ends up as a filter for that anchor label like TLnd0G8KR0.
If this is supposed to work, I'm on Firefox Beta with uBlock Origin and the built-in tracking protections.

[velvet fog]

in the todo list

[velvet fog]

@worthy cape deep link done

[worthy cape]

That's quick, nice!

[grave wren]

Do I misremember or did someone here do this and got told off by support? https://www.reddit.com/r/pathofexile/comments/iatj6w/poe_crafting_assistant_a_new_tool_to_help_with/

As in block overshooting on crafting mod hit

[worthy cape]

Seems familiar

There's a lot of dubious ideas passing through here tho ^_^

[grave wren]

Yeah hard to keep track but it was a hard no for things like that I seem to remember

[grave wren]

Ah interesting

[hazy fiber]

IIRC the game doesnt tell the server if it copies and item to clipboard so thats not a serverside action, though yes that tool still violates the "every time with no condition"

[worthy cape]

Fairly sure I mentioned it back then too, but it sure violates the spirit of tooling and the game.

[velvet fog]

Our plan is to change the way that Path of Exile interacts with the Steam patcher before the release of 3.12, which should fix slow patching. We'll post more information when it's available.

[worthy cape]

Indeed, that'll be fun.

[worthy cape]

E-hugs to Omega for the PyPoE stable.py, consuming the JSON flavour of it from C++ now.
https://i.imgur.com/YvO7PNS.png

Some day I'll remember that I need to use a JSON library with objects that report members in file order, DAT parsing gets Really Confused when you scramble the field order ^_^

[simple ravine]

we think alike, @worthy cape https://github.com/andreandersen/PoeSharp/blob/master/data/update-spec.py

[worthy cape]

@simple ravine Using the laziest approach to representing data read out of the DAT files. JSON objects 😄

Way more suitable for it than one'd expect.

[simple ravine]

u mean like RePoE, zao?

[worthy cape]

@simple ravine Internally in my code.

nlohlmann::json dat_table::get(size_t row, size_t col);

ref|list|ulong ends up as a JSON array of integers, etc.

I needed some discriminated union/variant type to represent the values I could get from a DAT table, and it turns they map "fine" into the in-memory type that the JSON library happens to provide, instead of rolling my own.

This lets me get on with the actual thing I want to make, which is what's important.

Should probably fix some out-of-bounds possibilities too, specs are a bit too volatile.

[rugged night]

What is the easiest/suggested way to (programmatically) get a list of league names as they happen? I know I can grab them from the TradeAPI... but thats a lot of "useless" data if all I want is new league names

[worthy cape]

http://api.pathofexile.com/leagues?

Or also private leagues?

[rugged night]

Private leagues too, if possible?

[rugged night]

Yeah... I found leagues by parsing the TradeAPI and ignoring all the item data... was just hoping there was a better way

There's an example on leagues API : http://api.pathofexile.com/leagues?type=season&season=Medallion&compact=1

...I tried changing "Medallion" to "Harvest" and it didn't work though

Surely thats from back in 2016?

[cosmic saffron]

it's the same stuff you'd get from accessing your account's seaons&events panel

[golden bane]

@worthy cape Hi, is there any reason you unpublished your poe-rs repo?

[worthy cape]

@golden bane I don't like having stalled projects visible as someone might try to use them while in a volatile state. I started moving the format documentation over into a separate book-style repo but not sure what state that's in.

Anything in particular you're looking for?

Was a bit hampered by trying to do 3D graphics and UI in Rust, where the Dear ImGui bindings are in a bit of a flux and doesn't do docking/tables; GL code is a bit cumbersome to do. It's weird in C++ now where I can just write the code I need to write, instead of recasting it into what the bindings might suggest 😄

sqlite3's transaction handling with lifetimes pushed me over the edge.

[worthy cape]

(made it public again until I get going on the book repo)

[golden bane]

@worthy cape I was mainly looking for the docs, thank you

You could archive your repo to indicate that it is unmaintained. I couldn't find anything about unarchiving a GitLab repo, but at least on GitHub you can revert the archive status if you happen to work on it again

[untold relic]

Hi guys, I've been doing poe app where you can check eq and characters on account. After fetch API i get response:
Access to fetch at 'https://www.pathofexile.com/character-window/get-characters?accountName=Anavrinsic' from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
Now it's working only on localhost cus i use proxy, but after upload on firebase it's doesnw work. Anyone have idea how to fix it? I'm working in ReactJS.

[untold relic]

@earnest radish thank you! i was fighting with this through last whole week, everything works now

[worthy cape]

I wonder at what point I'll have to abandon regexes in state machines, some file formats are getting gnarly: https://gist.github.com/zao/b2f1b285172dee9c4360fc595081e7a2

[simple ravine]

@worthy cape, I used parsing combinators to parse these IIRC

https://github.com/Geal/nom

for Rust

Kind of build things up lego-esque

https://github.com/Geal/nom/blob/master/tests/ini.rs
INI parsing using parsing combinator

Here's my .ot file parser, as an example:
https://github.com/andreandersen/PoeSharp/blob/master/src/PoeSharp.Files/Ot/OtParser.cs

[worthy cape]

This is C++, so dreading looking at things like Boost.Spirit or alternatives.

In poe-rs I used nom a fair bit, but it got a bit cumbersome once you started branching on file versions.

Loops were also weird, with a lot of explicit input threading.

[simple ravine]

Ah, you're holding a bazooka to your leg.

Perhaps treat the versioning as a normal branching in the combinator?

[worthy cape]

Sure, but it's quite intrusive and manifests like "field 5 of line type X is not present in versions < 25", representing it all in both the resulting data and the parser is a royal mess.

[simple ravine]

what file is that again? aot?

[worthy cape]

The one linked above is ao/aoc

[simple ravine]

do you have an example of the previous version?

[worthy cape]

Demonic files are ones like ARM that has 29 versions (thus far).

[simple ravine]

Demonic? ARM? I'm confused 🙂

[worthy cape]

For AO/AOC I'm mostly hampered by that I don't know what the possible statements are.

[simple ravine]

for me ARM is 'Azure Resource Manager' json files describing Azure resources for Infrastructure as Code demployments 😄

[worthy cape]

Like in the one I linked, I parsed the brace-sectioned "members" into a dictionary from string to string/bool.

Then I ran into the single file that also had the "animation" statement, which suddenly makes the parser have to read in those indented animation events.

I either get to look at all files ever made and guess at what the schema is, or try to represent it in a generic syntax tree.

[simple ravine]

I would suggest a 2-step abstraction approach, where you first try to find a generic syntax tree which you represent somehow, and then use that to further make sense of it all

if you can use indentation levels that'd be one way perhaps

[velvet fog]

Im using devil preg '/(\S+)\s*\{((?>[^\{\}]|(?R))*)\}/is' on ot parser

[simple ravine]

ok yeah, i guess it depends on how 'semantic' you want the data parsed into

[worthy cape]

I've been postponing it for a bit, but this is probably a good time to write the outlined GGPK loader, so I can sift through all the old files.

[simple ravine]

🙂

[worthy cape]

Also need to move all the info panels out into a separate project that builds faster.

[simple ravine]

On another note, I am starting to formulate an idea for a project to keep my programming instincts alive

I might be taking on more than I can chew, but that's half the fun

I want to make an application that is geared towards intermediary & advanced users who are flipping currency.

The idea is to allow the user to compose different pieces of information based on their goals and needs. An example would be if a user wants to flip exalts => chaos, chaos => alts, alts => exalts etc, e.g. 3-way arbitrage

Composing in information from trade sites, poe.ninja-esque sites, their own stashes to determine how much of a currency they have left etc

And using a node-based system to help them compose this, similar to this:

It's going to be glorious

[grave wren]

Does it work for ssf?

[simple ravine]

[simple ravine]

stuff like that perhaps

possibilities should be many

if one then could componentize these by promoting some of these inputs to one-level-up inputs, you could create components and reuse them, and create crazy combinations

[worthy cape]

Imagine programming, but bad 😄

[simple ravine]

lol

"rapid application development"

[mortal bone]

@simple ravine that is a pretty simple graph problem lol you can use Bellman-Ford to find loops in your graph. Then you can just hook up the output to a nice little node graphic like you see there

[simple ravine]

@mortal bone pretty much yeah

in Houdini, you can 'promote' paramters up to parent graph in a sub-graph, and drive it from another place and reference

the only thing I thought of was just what you touched on, circular references that needs a little bit of protection

[worthy cape]

You should see the noodle soup that is PoE materials 😄

[simple ravine]

node-based materials is the new black

this is my Octane material for an earth surface

[worthy cape]

https://i.imgur.com/z4WNtyj.png

Sorted one: https://i.imgur.com/KwQGlqK.png

[simple ravine]

looks pretty straight forward

[worthy cape]

Biggest bother is figuring out how the renderer underlying it all looks, and what of those are exposed results and which ones are basic blocks.

Solved the problem by shaving some yaks instead.

[simple ravine]

Well, procedural system

you kinda need the parameters for each node as well, and the logic behind them

[pine jolt]

Hey everyone, I wanted to post this here for visibility to other providers, if you're interested please comment: https://www.reddit.com/r/pathofexiledev/comments/igg532/round_table_for_community_tool_developers/

Reason for Post:
PoE is quite overwhelming for new players and individuals who take a break for a league or two sometimes struggle to catch-up. playing this for 6 years i can only imagine what its like for a new person to come to poe to be overwhelmed by that and then further overloaded with so many tools. We may be able to reach a great outcome if we work together providing less confusion, more robust tooling, and reducing technical issues.

Possible Goals:
Gauge interest in other similar projects combining teams
Discuss possibly making a platform (may be a bit of a lofty idea considered projects are in different languages) - at minimum teams who are part of the collective could all use same agreed upon nav bar in header
Reduce backlog, increase time to delivery by having more people on same team
Mitigate pain-points utilizing various individuals backgrounds, ie. UI/UX, net ops, sysadmins, CI/CD, DBA
If there is a collective, instead of direct contact for possible similar issues, we could utilize gitlab or some other issue logging system and have a couple individuals look at submitted issues, do de-dupe and add submitted issues to project issue boards

Who Should Consider Attending:
In my opinion this would be all tool providers, whether you have a web app/site, executable, auto hot key script or discord server providing services. Again, goal is to provide for less confusion for community and work together towards a common goal

Future Tooling:
If such initiative is successful, those who plan to develop additional tools would be most likely to reach out to be included in a project. If they already have a tool there could be a submission form in nav

There are obviously more topics we could discuss, hope this piques your interest

sorry for the massive dump, wanted to provide relevant info in a single post though

[simple ravine]

There have been websites that curate POE tools, I cant remember them right now though.

If you want to be a part of the community, this channel is a good place to start. Several of the tool developers are somewhat active here.

[pine jolt]

yessir, wasnt really after a curation site

[simple ravine]

Leveraging different talents, I'd watch with interest, but realistically I think it'll be difficult. Maintaining one's own project is normally enough (or sometimes even more than that). Additionally, the skillset is fairly spread out between different tech stacks.

[pine jolt]

agree with all of those points. as an example on leveraging the expertise of individuals, theres a couple well know sites that either have api or db related bottlenecks/potential load issues which turns people away from their tool to another medium either temporarily or permanently. if they could leverage the correct people those issues could go away, or happen less frequently. but ultimately, if similar projects wanted to combine and put all their great ideas together this would allow for more robust tooling, sharing of ideas (can only get so far with just one mind on a project), and ultimately allow for a more streamlined out-of-game experience for the end users so they will enjoy playing much more

[simple ravine]

I am struggling to visualize what you mean

When people have issues (when developing) with something PoE related, they come to this channel to discuss it.

[pine jolt]

that's operating under the mindset that we stay with all the disparate tooling which ultimately confuses end users, also if similar teams combine then the best features of the various tools can be implemented. as an example, i love some features of one trade macro but like the additional analytics and UX of another

[grave wren]

sorry but what are you trying to sell

[pine jolt]

not trying to sell anything

but i did do a large post above with this link as well as a synopsis: https://www.reddit.com/r/pathofexiledev/comments/igg532/round_table_for_community_tool_developers/

[simple ravine]

I think he's trying to standardize the way tools are developed and used, maybe like that wolf-something platform?

[pine jolt]

i guess step one is to gauge interest, if enough people are on board then can move forward

[grave wren]

yes i read that and it focusses a lot on what endusers might want, not why people write tools

[pine jolt]

but ultimately arent the tools being developed for the end users? i know there are hobbyists but not everyone falls under that category

[simple ravine]

99% of the developers are doing at as hobby, some are doing it for the fun of it, if some people use it, hey enjoy

[pine jolt]

yessir, and those individuals obviously wouldnt show interest but for those who are interested, the above information would be for them

[simple ravine]

i have to deal with scrum masters and product managers at work, dealing with standardizing enterprises' bs, i don't need it when messing around at home

that's me though, perhaps there are others who see it at another light 🙂

[pine jolt]

i definitely feel you on that, doesnt mean we need to implement a scrum master, thats why im trying to setup a round table to see what people are interested in and their boundaries

[grave wren]

that is my takeaway too you're trying to sell overhead on my hobbyist projects and i dont see too many benefits for it

imo you need to focus on what devs get from it

[pine jolt]

again, its a round table, all topics are up for discussion, im just doing my part to try improving the experience for all

im not heading this meeting, just laid out a couple topics of discussion, per thread i mentioned another individual would be better to lead, just trying to start a ball rolling in a positive direction

[simple ravine]

the issue I am having to get sold and feeling the desire to jump on that wagon is the the value proposition being quite abstract, and the solution being 'lets discuss'.

[worthy cape]

I skimmed through the thing when seeing it in "another discord", didn't see much of interest to me as a standalone developer not using the API or databases.

[simple ravine]

just some feedback that it might be worth thinking about it once more, and come back with it a bit more thought-out - no offense.

[grave wren]

you basically said what i wanted to point out more concise

[pine jolt]

The possible collective would consist of individuals from many backgrounds and viewpoints, i believe what you mentioned is an impossible ask or a post so long that no one would read. again, i'm not trying to 'sell' anything, for individuals who are doing it as a hobby and the user experience isnt top priority then this would not be for them. constructive criticism would be helpful rather than all the reasons why this is a bad thing that we as a development community should not work towards

¯_(ツ)_/¯

[worthy cape]

I don't quite get what this is. Is it supposed to be a discussion forum, a hosting service, shared infrastructure, design guidelines?

[pine jolt]

did you read the reddit post?

[worthy cape]

We kind of have the community thing already here.

[simple ravine]

yeah, I did and I am confused

[pine jolt]

what should i clarify to make better impact?

[worthy cape]

Skimmed while working.

[pine jolt]

no immediate response is necessary so feel free to read when you have more time and let me know if i should re-word things or provide additional clarification

i mean, this is a lofty goal but in my opinion, one worth entertaining

[simple ravine]

Your idea is too abstract. I can't really visualize what you're proposing, other than a collective helping with each other's projects, and potentially sharing functionality between them

[pine jolt]

thats partially accurate, i mentioned in the post that similar tools may be further ahead combining projects as well to reduce confusion and per message above, best of all similar tools that makes one tool someones favorite over an other can be implemented thus making for an even better tool for the community

also, projects in the collective could be part of a 'platform', which is tbd which is why putting heads together would be useful. i see it as projects that are part of the collective could all use the same nav header in pages for flipping between projects, so with said nav bar and combined projects, individuals experience with what we make will be more streamlined

[simple ravine]

While that is likely a positive impact, if it would work. However, lessons learned from a couple of decades of open source history, tell us that there's a flaw. We're all humans.

[pine jolt]

i dont follow the 'from a couple of decades of open source history, tell us that there's a flaw' do you mean end users submitting issue via github?

[simple ravine]

A lot of the projects (if not all of them) are open source, and there are quite few contributions / PRs as-is.

[pine jolt]

i understand that, again.... possibility of combining tools, not the 50+ disparate tools available now

[simple ravine]

As in most OSS projects are 99.99% consumed by others, however there are very few contributions back to them. People are busy with their own stuff, or find that the goals and wants of the developers are so different that it warrants a different project, hence we have 'competing' apps that are doing similar or identical things

There have been similar discussions in here before a couple of times, not resulting in anything as of yet.

[pine jolt]

i understand that, i was making a competing crafting app for a long time until i started working on a tool that was more useful imo for the community

at the time i assume there was no interest in such a proposal?

or was there just no direction?

[simple ravine]

I think that direction isn't really the part that is the problem

[pine jolt]

so its primarily people just want to do their own thing when they have time

[simple ravine]

That's my current understanding

[pine jolt]

k, well we'll see if there's interest, if there's not then there's not. all i can do is hope for the best

[simple ravine]

Here's some constructive criticism (in my mind), no offense intended.

Reading the pain points again, that you're putting as examples are likely not that common when it comes to PoE tools, except a few rare cases, such as poe.ninja (and knowing rasmus has not that much time, and not really interested in open sourcing). Most are desktop tools, AHK scripts etc.

The language is quite enterprise-y, using words such as 'time to delivery' or 'time to market' isn't really a major concern for most, I am guessing.

[worthy cape]

My pain points: GGPK files are large, disks are slow, bulk reverse engineering tools don't exist, library ecosystems are bleh.

[simple ravine]

Possible platform for organizing a large percentage of the tools to make for more streamlined access while individuals are gaming

This part is confusing me a lot.

[grave wren]

My pain points: GGPK files are large, disks are slow, bulk reverse engineering tools don't exist, library ecosystems are bleh.
@worthy cape get more ram

[pine jolt]

also when using pypoe some further search n replace regex is needed

[worthy cape]

Assuming I ever ship, the most I need is infrastructure for self-updates, patching, etc.

[pine jolt]

so having shared baked data would be nice, since i have a few servers i can put heavy load on thats one example of help

[simple ravine]

If there’s not an individual in the collective who has a specific background or time to do changes we can the source such individual from contacts or reddit. Support is assumed to be a task across the board so we could potentially create a ticketing system for issues and find individuals to sift and add to issue tracking system whether it be gitlab etc.

Again, hobby programmers aren't really looking to outsource problems to contractors

[worthy cape]

@grave wren I could fit all GGPKs ever made in like five compute nodes at work, but ingestion would take a while 😄

[pine jolt]

This part is confusing me a lot.
@simple ravine i mentioned multiple times above about the navigation bar, again a topic for discussion

balls, quoted that wrong

[worthy cape]

I'd also get a whipping for occupying a quarter of the 3TB largemem nodes.

[grave wren]

nobody has to know

ask for more ram on the nodes while you're at it

less nodes used = less suspicion

[simple ravine]

That implies people want to do that, and are at the same time using a tech stack that and frameworks that are compatible, which is likely going to be a huge mess. Just saying, out of my background/experience in the development field.

[worthy cape]

I'm tempted to rehydrate the whole dataset on the new cluster storage when we get that, just to have some baseline load 😄

If it wasn't super sketchy legally, I'd publish a hash-addressable storage of all the files ever.

Now that'd be useful a whooping three people 😄

[simple ravine]

😄

[pine jolt]

right, i mean no offense but it seems that things are still being nit-picked apart. i can tell that even if i were to rephrase this that you personally would not be interested. per the mention of phrasing being to enterprisey, imo that shouldnt make a difference, yes its a term for enterprise but should i use lengthy descriptions when people either already hear that phrasing in the workplace or can extrapolate what is meant by it?

[simple ravine]

I think you have good energy, but probably need to direct it in a different way.

Let's just put it like that

[silent tree]

i think overall people are not sure as to why any of it is necessary

[pine jolt]

lol k

[simple ravine]

hse put it even more concise, faust

[pine jolt]

right, because everyone is enjoying their hobby projects

[worthy cape]

A lot of the infra is sufficiently non-bad. Hosting and ticketing is good enough and authors have enough control.

CI? GHA and CircleCI's decent.

[pine jolt]

ok, i guess i'll just leave discussion alone and see if people show interest on reddit. if anyone needs help with loadbalancing, k8s, server help lmk in the meantime as that was on of the things i could offer the group

[simple ravine]

Right, I think there's gonna be the regular huge debate on which X is best because of Y. Like, me personally prefer Azure DevOps and GitHub, and wouldn't want to figure out Gitlab or Bitbucket, while some feel the complete opposite.

[pine jolt]

meh, gitlab is same just has a lot more features

[grave wren]

the idea is commendable but it's too many words for basically this community with maybe more merged projects

[simple ravine]

I think another issue with that part is trust issues, like - knowing that you're going to actually host the thing properly in your cluster rather than rolling their own etc

[worthy cape]

@simple ravine Bitbucket, good joke

[silent tree]

something something cluster jewel rolling joke

[worthy cape]

Having migrated several repositories away from the Bitbucket Mercurial apocalypse, losing a hosting platform suuuuucks.

[simple ravine]

aye

And then theres the discussion of AWS, Azure, GCP, On-prem

I'd bring popcorn 🍿

[dense shore]

GemCutter's Prism?

[pine jolt]

thanks for the positivity @grave wren, i feel less frustrated now

[worthy cape]

Google Cloud Pony.

[pine jolt]

lol kroo

[grave wren]

dunno i just wanted to nudge you into being more concise

because anything you try to strap up is basically selling an idea

[pine jolt]

ewww gcp or aws for burning money

[silent tree]

i think one of the main issues is that a change in hosting won't really affect one of the biggest bottlenecks which is the api itself

[grave wren]

maybe we can sneak one of us into ggg and just fix the river

[silent tree]

like ur hosting can be as lean and efficient as u like, if the api isn't giving u anything back it's not made any difference

[grave wren]

in one quick sweep

[pine jolt]

you could distribute the load and proxy requests to containers or small vms

[simple ravine]

unless you get to sit next to their rack in texas

you can shave a few milliseconds

😄

[silent tree]

didn't we get viper into ggg for that reason foost

[worthy cape]

brb flying to NZ

[silent tree]

and however u manage the load it's still got to go to ggg servers

[grave wren]

wait i thought that was openarl

[silent tree]

and those requests are still ratelimited

[simple ravine]

@pine jolt I think that's frowned upon, circumventing the throttling

[silent tree]

ah ye i misread that

[pine jolt]

ah, havent read the ToS on that

[silent tree]

yeah i think that would be edging a little close to tos

to an outsider it could pretty easily be seen as trying to set up a dos

[simple ravine]

the rate limit is there for a reason

[dense shore]

Openarl was caught 😦

[simple ravine]

and hired 😄

[silent tree]

we saw why the limit exists last league as well when dsc/dhc couldn't trade at all

[worthy cape]

(not gonna lie, I'm still kind of pondering applying, just a big life upheaval if I do)

[dense shore]

Wasn't there something about spoofing the limit rate with that incident?

[pine jolt]

when i applied they were only hiring for those in NZ, now that they allow remote for some things i may entertain again

[silent tree]

err i don't remember

wasn't some livesearch related tool ddossing it

[worthy cape]

The big DoS was from end-user tooling with poor error handling, wasn't it?

[simple ravine]

i recall a couple in here encouraging me to contact them, but i think im fairly happy where i am right now

[silent tree]

yeah zao

iirc it was just relaunching requests and doing thousands at once

[worthy cape]

About as clever as sshfs.

[silent tree]

sshfs

[worthy cape]

(when sshfs loses connectivity it forgets all the credentials it ever has had and retries auth with no creds and no delays)

[simple ravine]

great success

[worthy cape]

fail2ban ftw

[golden bane]

@pine jolt (Just read though the conversation now) I think you generally miss the point of these tools being a no-strings-attached gift to the community

[worthy cape]

Fun fact of the day - sqlite3 blobs can only be 2 GiB in size, despite the existence of functions like sqlite3_bind_blob64 that may lead you to believe otherwise.

The biggest file ever in a GGPK is around 422 MiB.

[worthy cape]

Bonus points, sqlite3 has concurrency limits, resulting in SQLITE_LOCKED on pretty much any call, including statement preparation.

Should probably set up a gosh-darn postgres.

Unless someone has recommendations on idiot-proof low-latency KV stores?

[simple ravine]

Cassandra?

oh, idiot-proof, well... Docker to rescue?

[worthy cape]

My infra is ... not exactly container-friendly 😄

(despite reportedly working as a sysape, my home infra is in shambles)

[simple ravine]

Cobblers' children are worst shod. 🙂

[worthy cape]

^_^

[grave wren]

Just add a smoll docker node can't be much worse as is

[simple ravine]

inb4 one realizes cassandra should be a cluster of cassandra nodes, and down the rabbit hole one goes.

k8s, helm charts and all the glory

[grave wren]

Sounds like a fun endeavour

[simple ravine]

1 month later... 😄

[grave wren]

Isn't it all about the journey 😄

[worthy cape]

I tried building RocksDB as a submodule. Facebook really likes to bulk out their projects, don't they?

[simple ravine]

@worthy cape i need ur linux knowledge

I am trying to get enough space on / to do a distro upgrade

or is there another way?

It's Ubuntu 16.04 currently

additionally it feels like there must be a bunch of crap in there

i tried apt autoremove and clean etc, removed all old logs and /tmp

[worthy cape]

If it's a virtual disk, you could possibly enlarge it and grow the partition if it's the last one on the drive.

Did you clean apt caches and whatnot?

[simple ravine]

that's what u do with apt autoremove and apt clean right?

[worthy cape]

I like ncdu to get a feeling for what takes up space on a box, easy to drill into directories to see what consumes space.

[simple ravine]

ah ncdu is much better than du with a bunch of arguments

thanks

[worthy cape]

I've been parsing more AO/AOC now. The format is reasonably regular but people seem to be hand-editing the files and screwing up indentation.

[simple ravine]

ouch

[worthy cape]

There's stray whitespace before/after some lines, and also more interesting problems like this: https://gist.github.com/zao/9a507562eb81204bc60dc564194f3ec2

You already have to kind of rely on indentation to delimit the animation parameters, unless you start looking for the first token being a floating point value.

Another favourite is unbalanced string literal like "omg wtf bbq""

[simple ravine]

what surprises me is that they have some kind of parser that can deal with these human mistakes

[worthy cape]

I haven't mapped out what versions this was in, so it's probably actual bugs that either happened to parse or were for objects rare enough that they only sporadically crashed the client.

[simple ravine]

well I think I saw the some indentation issues in the Translation files as well

unless I made those mistakes myself

[worthy cape]

Just now ran into something that either requires me to know a schema or rely on indentation.

WindEvents
{
    continuous_effect = "wake vortex"
        size = "100.0 100.0 100.0"
        duration = 5.0
        is_persistent = true
        shape = "Wake"
        is_attached = true    
        bone_name = "root"
}

I'm leaning toward treating the bugs as bugs and only parsing the "correct" interpretation.

Need to wire up lookup from GGPK name to actual game version to get a feeling for what era these are from.

[simple ravine]

right, question is if the indentation after the first continuous_effect is correct or not

[worthy cape]

This resembles the kind of subobject that animation is for AnimationController.

Worked the last few days on getting my shotgun parser infra up, now I can run parse attempts against all files ever, deduplicating by content hash if they're location independent.

15M AO+AOC files over time, 50k unique payloads.

[simple ravine]

shit.. I think I might be screwing up this server big time now

[worthy cape]

\o/

[simple ravine]

i ctrl+c:ed sudo do-release-upgrade when it said "Press ENTER to do this thing"

just to check one last thing

and when I did it again, it said "Install all pending updated for your current distro version first"

so I did apt update && apt upgrade

and now its installing packages from 18.04 (I'm on 16.04 on this)

it's gonna blow up so big time

[worthy cape]

It has probably altered the release in /etc/apt/ files, might be a bit too late to undo that.

How's your backups? 😄

[simple ravine]

kek

it's just a hobby server (seed box)

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic```

well shit

I guess we'll find out soon

these messages doesn't look too good

well Plex etc is still alive

I just never reboot this

Linux, I curse you!

@worthy cape what option is the safest bet?

grub is some kind of bootloader or something right?

[worthy cape]

Yeah.

Look at the diff, then clobber if the changes seem acceptable or try to integrate your local changes.

/etc/default/grub is what you edit to alter the kernel command line and other user-facing configuration knobs.

update-grub processes that and other files, generating the actual grub.cfg in /boot.

[simple ravine]

feels like they made some changes that might be wise to take

ncdu did the trick

[worthy cape]

I use spacesniffer now for windirstat kind of things, I forget why I changed but I think it was due to crashes/perf.

[simple ravine]

i should probably build some simple box and host at home rather than renting one in netherlands for this

I spend €62 per month on this server, and it's starting to feel like a bit of an overkill

rented it since 2017

deluge, plex, sonarr, radarr

it streams media stored locally, yes

then we use the plex app to stream it to a tv nearby wherever we are located currently

my home internet is 1 Gbps up and down, same with the rented server

the main benefit is that any network traffic incurred by the software installed on the server (deluge and plex) will not interfere with my day-job at home

but I could always limit that activity to let's say 100 mbit/s during working hours

what's buffer bloat?

the basics

that's quite unfortunate

but i presume buffering is a way to maintain throughput when there's high latency

i.e. trade-off

ah nice

I was thinking of a NUC of some kind and a "dumb" NAS

potentially due to high CPU usage generating the random packets?

if your CPU is pegged at 100%, it'll affect everything else

try https://fast.com

and click the more info thing

[worthy cape]

Can't wait for BBR to ship more widely.

[simple ravine]

BBR?

[worthy cape]

New fancy congestion control algo.

https://www.ietf.org/proceedings/106/slides/slides-106-iccrg-update-on-bbrv2-00

On a more dev topic, I found out that postgresql connections are also not thread-safe, so I wouldn't gain too much by moving to postgres.

Shelved that idea and made a half-arsed connection pool in my code instead, spinning on busy.

[simple ravine]

the connection on the client side?

the server side should be safe

[worthy cape]

Yeah, libpq connections.

[simple ravine]

otherwise it'd be a useless database heh

[worthy cape]

Server side seems to be a bit limited in connection count too, recommending bouncer processes and whatnot.

[simple ravine]

connection pool is ur friend

open up N connections, put them in an object pool

rent them out

[worthy cape]

Went slightly more efficient there and gave each runner in a thread pool a separate pre-created DB connection, so it's more like thread-local storage instead of the synchronization of a shared pool.

Got a nice VFS interface in front of the different kinds of GGPKs I have now too, so I can use the same kind of operations regardless of if I have a regular GGPK, an outlined GGPK + database, or even a file system tree.

It should (hopefully) be adaptable to whatever the upcoming pack changes are too.

[simple ravine]

create a driver for <insert operating system>? 🙂

[worthy cape]

I've built both FUSE and ProjectedFS drivers... sketchy AF and has some annoying requirements.

Need to focus a bit on parsing the things I actually need for my project and leave the rest as caveats in format docs 😄

[simple ravine]

hehe

on another note, the 16.04 -> 18.04 upgrade actually worked after all

[worthy cape]

20.04 next then ^_^

[simple ravine]

now, 18.04 -> 20.04

had to do the -d switch on the distro upgrade

hopefully that's not gonna mess things up

[worthy cape]

I like the official upgrade procedure for Linux Mint. Paraphrased - "back up your stuff, pave the machine. Don't bother anything else."

[simple ravine]

hehe

i have had this one since 14.04

14.04 to 16.04 was a nightmare

a lot of things changed

had to vpn into internal network of the data center and kvm myself into the box

ok here we go

[simple ravine]

it is always python messing up, isnt it

[civic crane]

.dat structural change chance:
change field type - 1%
append field - 95%
insert field - 1%
remove field - 2%
reorder fields - 1%

@obtuse citrus can you confirm these numbers from experience maintaining pypoe spec?

[obtuse citrus]

I didn't run statistics, but it's mostly appended fields yes or just new files entirely

type changes and removals occur often enough to be annoying though. changes often are something like a single reference to list reference to achievementitems when they add new challenges

[simple ravine]

hilarious side-note of updating to Ubuntu 20.04 on the server

it automatically suspended the server after 30 minutes of "inactivity" rendering it unresponsive.

took a while to figure out that was the issue. ugh

[worthy cape]

Found that at work, great fun.

(tends to be sucked in if you install a wee bit too many desktop-ish packages)

[simple ravine]

i wanna try to figure out how to get rid of it all without wrecking the whole system

[worthy cape]

Mask the systemd units, IIRC.

[simple ravine]

yeah I did sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

some reported it didn't help, so went into gnome config and removed stuff too

now trying sudo systemctl set-default multi-user as well

now seems to be stuck here:

but SSH works fine

so I guess I'm happy now

[worthy cape]

@simple ravine All we do is https://gist.github.com/zao/cbf320599572094673d1032965d2e118

[simple ravine]

404

[worthy cape]

Guess it's not public-public 😄

[simple ravine]

grab the relevant lines?

[worthy cape]

So yeah, just mask those four.

Ok... who decided to add comments to AO files?
https://gist.github.com/zao/13d9cb2fe00a497b3332ec57c04083be

[simple ravine]

LOL

[worthy cape]

https://i.imgur.com/1Zrsezq.png

I love when my tooling is starting to work.

[simple ravine]

zao, any idea how to get past this?

sshing into the box works fine

[worthy cape]

There's some handover of consoles between early and late boot, you might've scuffed the configuration somewhere or disabled the TTYs or something.

Especially if it's a serial one.

[simple ravine]

this one used to boot into the GUI

then according to some guide, I was supposed to do sudo systemctl set-default multi-user

ah, had to press ctrl+alt+f3 and it sent me to the login

[worthy cape]

Good news, only the asset viewer template has comments.

[worthy cape]

OTs sure look "fun".

[simple ravine]

huehue

[native zephyr]

Currency Cop Version 2.0.0-beta.20 (Tabre's Fork)
Release Notes:

• Added chart for historical data
• Merged in tiered map pricing. Thank you Troy Wolters (TW80000)
• Added efficiency meter to show Chaos per hour based on the data in chart
  Calculated as follows:
  ( C[n] - C[1] ) / ( T[n] - T[1] )
  Where:
  - C = Total Chaos
  - T = Timestamp
  - n = Number of observations
• Added support for character inventory as tabs

https://github.com/tabre/currency-cop

[worthy cape]

@golden bane Got my arse in gear and set up mdBook CI for my repo - https://zao.github.io/poe-doc/

[golden bane]

Looks great, thank you!

In the meantime, I wrote a GGPK reader for fun

It's in Python, but a lot faster than PyPoE

[worthy cape]

I've been meaning to see if I can get PyPoE to understand my outlined GGPK indicies and see if it speeds it up, but so much else to play with.

My Index.ggpk is like Contents.ggpk but all file contents are stripped and thus the payload sizes are zero, instead having an external common KV store from SHA256 digest to payload.

The added locality in file cache seems to do wonders for my bulk parses in native code.

[golden bane]

PyPoE doesn't read in any payloads until you want to view/extract one iirc

[worthy cape]

Yeah, I'm mostly concerned that the slowness in initial reading is due to the size and spread of the trees in the file.

[golden bane]

Seems like the initial read still takes up the majority of the time even after the file is cached by the OS

(the light green one is file reading as well)

[worthy cape]

The index for current patch is 57 megs, for example.

If you want to play with one: https://zao.se/~zao/poe/ggpk/Index-1176132989565732489.ggpk

[golden bane]

This is where I am right now. Interestingly, decoding the UTF-16-LE strings to UTF-8 takes up the single most time

[golden bane]

Parsing for both is probably higher on Linux btw, provided you use memory mapping, as you have mmap.mmap.madvise(mmap.MADV_SEQUENTIAL) available

[candid ridge]

If I wanted to grab a copy of the racing gauntlet ladder is there an easy way to do that? There are 15000 entries.

http://api.pathofexile.com/ladders/Racing+Gauntlet+Points+Event doesn't work

http://api.pathofexile.com/ladders/Racing+Gauntlet+Points+Event+(PL10667) this works

is there a way to grab all 15000 entries without doing 200 manually at a time

[golden bane]

@candid ridge No, you have to fetch 200 at a time or read from the .csv export

[candid ridge]

@golden bane how do i grab a csv export

https://www.pathofexile.com/ladder/export-csv/league/Racing+Gauntlet+Points+Event+(PL10667) this works

is there a way to get past the first 2000?

https://www.pathofexile.com/ladder/export-csv/league/Racing+Gauntlet+Points+Event+(PL10667)/index/1 this doesn't work

[simple ravine]

@candid ridge Did you figure it out, or shall I poke at it real quick?