no, it added in this patch. maybe for macOS
#tooldev-general
velvet fog
timber path
Then I'd expect them to be present for the English dat files too... But they're not, so that makes me doubt it's for macOS
velvet fog
it present for the english in Data.datl_4.bundle.bin
simple ravine
On another note, C# 9 oh lord thank you jesus
For anyone interested, https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-9 for a summary
worthy cape
What do you think of this?
typedef uint8_t* OozMem;
OozMem OozMemAlloc(size_t size);
int64_t OozMemSize(OozMem mem);
void OozMemFree(OozMem mem);
int64_t OozDecompressBlock(uint8_t const* src_data, size_t src_size, uint8_t* dst_data, size_t dst_size);
OozMem OozDecompressBlockAlloc(uint8_t const* src_data, size_t src_size, size_t dst_size);
OozMem OozDecompressBundle(uint8_t const* src_data, size_t src_size);
(last one not implemented yet, of course :D)
Could drive it without any major effort from Python CFFI f.ex.
simple ravine
C++?
worthy cape
C API for oozlib.dll.
simple ravine
oh
simple ravine
yeah
I'm still a little into the attempt of implementing it in .NET 5 (new .NET Core thing)
I'm not sure about the legality required tho
if I re-write it from scratch, if I need to still abide by the original concept's license
worthy cape
It's even more fun tho, some of the files (comp_*) are not GPL but some ad-hoc "educational only" shenanigans.
Almost user-friendly - https://gist.github.com/zao/40d03fb9234d071db9066bfef1d58c44
simple ravine
I'd suggest to abstract away the memory management
makes the API surface a little easier, less boilerplate
worthy cape
@simple ravine It's only there if you can't allocate your own buffers or if the size is unknown.
simple ravine
Caffeine pill taken, first .cs file created 😂
worthy cape
In those cases a slightly richer buffer type is required for the interface.
simple ravine
oh lord what have I done
Hmm, would be nifty with something that I could create some live unit tests with
hallow ice
hey people, i released a build for the skills overlay i posted on reddit a few days ago https://github.com/csiqueirasilva/poe-skills-overlay. having some trouble testing it on different resolutions
lavish gust
@hallow ice this is probably against ToS as flask duration overlay was confirmed to be
snow kernel
@hallow ice this is probably against ToS as flask duration overlay was confirmed to be
@lavish gust wasn't MercuryTrade's Overseer confirmed as something they don't like, not something against ToS?
lavish gust
possible, I don't remember exactly but probably because of such overlay they decided to implement it in game
hallow ice
i did a bit of research on that, and asked them before releasing it - they stated some things: "I would recommend refraining from creating or using any program that automates gameplay or does more than one action with a keystroke or mouse click as well as anything that interacts with the game client/files to provide an advantage over other players or provide information that isn't normally visible." - which I believe mirroring the screen back to itself does not count
also, the app runs on desktop mode, without the client - which is something posted in the past as a "rule of thumb" for safe apps
but i put a disclaimer saying to people to refrain using it, due to gggs volatile tos. sure they can out of the blue say this is not ok. so far, theres no evidence of that tho
snow kernel
ye if you did some say pixel scanning to detect buffs in the buff area and display them somewhere else it could be construed as unfair advantage
hallow ice
overseer initially tracked cooldowns and other stuff on screen. i think after the ggg statements on it, the creator added the screen mirroring to it, but later removed overseer from mercury trade entirely. only believe it was due to the tracking features, not the screen mirroring. but its only my opinion on the matter tho.
no additional processing is being done on the pixels, made sure of that
just reflecting on one point to the other
you think this would still be a total violation?
snow kernel
hey, lootfilters used to be a memory hack couple of years back, they change their mind on qol. buff/debuff management in poe is an absolute nightmare and cleaning it up would not compromise their vision for gameplay. but just to stay on the safe side, i doubt such discovery automation would be something they're absolutely ok at this point in time
hallow ice
might create a branch to develop it further, if some people become interested in the tool as it is
would like to change the c++ code on it to exporting png images to stream mjpeg tho. if anyone knows how to do this properly, would be great. got minimal c++ experience
worthy cape
That'll do, should be enough to build some tooling on.
I even commented the interface 😛
lavish gust
@hallow ice I could help you with it (but not today)
vapid pulsar
So regarding the oodle decompression. I have got a method working that uses the bundled version of oodle inside of PathOfExile.exe
Does anyone know if it's against the tos to mess with the exe file?
simple ravine
Modifying it normally is against it. Not sure about utilizing certain functionality inside of it.
ToS says something about reverse engineering though.
they to some extent turn a blind eye on that note, when tool developers utilize certain assets to create community tools?
vapid pulsar
Yeah i think this falls into that grey area. with tools dev. I might have to get an official determination of this method.
It involves injecting code into the exe to jump to the decompressor which might be too far in GGG's eyes.
balmy flame
decompression... so what about compression? 
worthy cape
Not quite sure what legitimate needs there are for recompression.
warm zodiac
hypothetically
if ggg dont want to tinker with ggpk they would add at least hash checks
worthy cape
That comes at a runtime cost for the players. Some integrity is verified in the patcher to ensure that the pack is in an expected state. Don't fuck with the game.
worthy cape
Ugh, forgot about the SAFE_SPACE.
worthy cape
That kind of ruins my current caller-allocates interface unless I make it part of the contract.
rapid pagoda
@worthy cape re. licensing on ooz -- I'm pretty sure that ooz is just a decompiled version of the oodle library with some names injected back into it 🙂
worthy cape
Got drop-in compatibility for future-proofing:
bun = bun_lib.BunNew(b"libooz.dll", b"Ooz_Decompress")
bun = bun_lib.BunNew(b"oo2core_7_win64.dll", ffi.NULL)
Did we figure out what the tail of the index past the file info was?
velvet fog
it has another struct and a bunble.bin inside index.bin
typedef struct {
uint32 name_length;
char name[name_length];
uint32 bundle_uncompressed_size;
} bundle_info_t <optimize=false>;
uint32 bundle_count;
struct bundle_t
{
bundle_info_t bundle_info[bundle_count];
} bundles;
typedef struct {
uint32 unk[2];
uint32 bundle_index <comment=BundleIndexComment>;
uint32 file_offset;
uint32 file_size;
} file_info;
uint32 file_count;
file_info files[file_count];
string BundleIndexComment(int bundle_index)
{
return bundles.bundle_info[bundle_index].name;
}
uint32 some_count;
struct some_t
{
uint32 unk[2];
uint32 inner_bundle_offset;
uint32 unk3;
uint32 inner_bundle_size;
} some[some_count];
struct {
uint32 uncompressed_size;
uint32 total_payload_size;
uint32 head_payload_size;
struct head_payload_t {
enum <uint32> {Kraken_6 = 8, Mermaid_A = 9, Leviathan_C = 13 } first_file_encode;
uint32 unk10;
uint64 uncompressed_size2;
uint64 total_payload_size2;
uint32 entry_count;
uint32 uncompressed_block_size;
uint32 unk28[4];
uint32 entry_sizes[entry_count];
} head;
struct entries_t
{
local int i <hidden=true>;
for (i = 0; i < head.entry_count; ++i) {
struct entry_t {
byte data[head.entry_sizes[i]];
} entry;
}
} entry_info;
} bunble_bin;
@velvet fog If you haven't decoded the inner bundle yet - https://zao.se/~zao/poe/3.11.2/inner.bin.decoded
Lots of string:y material.
simple ravine
inner bundle?
thick hawk
Seems like high compress levels are incredible slow.
oo2core_8_win64.OodleLZ_Decompress() vs unpacked file (502Kb -> 5.7Mb) = 86 ms, ok.
oo2core_8_win64.OodleLZ_Compress(), Leviathan vs 5.7 Mb file:
"Normal" level (4) == 784 ms
"Optional1" (5) = 3755 ms
...
"Optional5" (9) = 104 seconds (!) <-- lol'd
ooz:
ooz -z --leviathan --level=5 arc_03.bundle.decompressed.bin ooz_pack_Out.bin: ~3 seconds
--level=9: AppCrash on 5th seconds after launch 😦
velvet fog
.index.bin has another bunble.bin inside
// inner bundle of _.index.bin
typedef struct {
uint32 unk;
if (unk != 0) {
string str2;
}
} unkk_t <read=unkComment>;
local uint32 i = 0;
for(i=0; i<789052; i++) {
unkk_t unkkk;
}
string unkComment(unkk_t &unk1)
{
local string val = "";
if (unk1.unk > 0) {
SPrintf(val, "%d,%s", unk1.unk, unk1.str2);
return val;
}
return "0";
}
hmm
worthy cape
Ah, just reading until the end.
String table entries: 788072
simple ravine
Wish I was better at the reverse engineering craft
worthy cape
@velvet fog I wonder if this is misaligned in some way... unk1 has some weird wandering values: https://gist.github.com/zao/42db433fae43832a951f8cb874a81869
A lot of those bytes look like characters.
worthy cape
There's 43 instances of an unk1 with the value 0x2f747241 which is literally "Art/", which seems super wrong.
worthy cape
It desyncs rather early from the text, first hundred bytes or so.
@velvet fog The largest value of some_t::unk1 is 10922345 while the decoded inner bundle is 10933971 bytes in size. I suspect that variable is an offset.
worthy cape
Yeah, following those offsets puts you a few u32:s before path-like strings, where the pattern I've seen thus far is "one or more u32 with value 0, one u32 with value 1, then a string"
Time to sleep.
worthy cape
It's some weird common substring scheme, see this https://gist.github.com/zao/50eabba0812ad32fb9020136e41e8ad8 example entry sliced out from unk1 offset and unk4 size. unk3 is often the same as unk4 but sometimes much smaller.
The corresponding directory in 3.11.1g contains two files on_act_defensiveAttackLarge.amd and on_act_defensiveAttackLarge.ast.
Almost like 00 00 00 00 is "remember this location" and 01 00 00 00 is "append string on marked location".
I suggest that these are compact listings of directory contents.
worthy cape
Feels like a trie of sorts.
velvet fog
you just sleep..for 2 hours?
worthy cape
Nah, stayed up for two hours more, the trie line was an insight just as I hit the pillow 😄
fickle yew
Nice @simple ravine - did you finally merge new and master on PoeSharp? 🙂
simple ravine
Not yet sorry 😦
So much ADHD distractions 😂
Now there are bigger issues to figure out, @fickle yew ... They added a bunch of new stuff. Encoded bundles of files, seemingly a bit like russian dolls
@fickle yew which file type parser were you looking for to use? I could probably fix that today
It's a good push, so I can actually merge
simple ravine
Yes. Might have to use GPL if need to use the oodle decoding stuff
fickle yew
From the master branch what I'd need is probably PoeSharp.Files.Dat and PoeSharp.Files.StatDescriptions
simple ravine
Ok, cool. Dat is probably the most complicated to move, so then I can just move them all 🙂
fickle yew
But I guess it's also just easier for you to manage one branch instead of having to explain the difference every time 🙂
simple ravine
Need to figure out how to proceed with the bundle encoding stuff in between though
worthy cape
@velvet fog I believe that the inner bundle cannot be segmented without the some table from the index.
simple ravine
Instead of API surface -> GGPK -> Dat, it'll be API surface -> GGPK -> bundle decode -> dat
But I guess it's also just easier for you to manage one branch instead of having to explain the difference every time 🙂
@fickle yew
haha yes. much so.
velvet fog
this looks great
timber path
The purpose of this seems more like a ToC, only for GGG to verify if all files are present...? Doesn't seem to serve any other purpose?
worthy cape
They begin with some number of zero words, then text that kind of gets reused to form multiple full paths.
velvet fog
looks like the replacement of _.index.txt
worthy cape
I'm not sure if there's any mention of bundles in there, or if it's just a pure directory+filename listing.
I would guess that it's for enumerating directory contents, maybe?
grave wren
What's that tabbed terminal @simple ravine
fickle yew
Windows Terminal I'd guess.
At least that's what I've been using every day since it came out.
worthy cape
I had some trouble where my muscle memory operations didn't work as expected in a terminal, might've been select-to-copy or some movement keys.
simple ravine
right click to copy after select
do you normally use linux or mac, zao?
// If enabled, selections are automatically copied to your clipboard.
"copyOnSelect": false,
// If enabled, formatted data is also copied to your clipboard
"copyFormatting": false,
you also have those settings u can set
worthy cape
@simple ravine Linux at work, mostly Windows at home.
simple ravine
Linux on your desktop as well at work?
worthy cape
There was some other shenanigans with it that made me bounce off, maybe something with fonts or something. I'm fine with mintty.
Yeah, there's value in having a shared home directory with the clusters and all the Kerberos and other auth set up properly.
pseudo ocean
I prefer cmder tbh
pseudo ocean
yeah that's probably true
simple ravine
what is the thing that you prefer with cmder over wt?
pseudo ocean
I had some difficulties gettting tab complete to work which was the main thing, also i din't really like the popup on when you paste in a line with a new line character in it, or the fact it keeps adding the azure cloud terminal to your list of terminals even if you delete it.
Idk maybe its not entirely fair, cus i haven't extensively tried to solve any of these issues yet.
simple ravine
It has been in preview for a while, and gotten a lot of QoL fixes not too long ago. But I mean, if you're happy with cmder, stick with it
dull laurel
someone already decoded the new file formats?
worthy cape
@dull laurel Partially.
There's quite a lot of history in the channel the last few days, also https://zao.github.io/poe-doc/bundles.html
dull laurel
just checked the folder after the update. lots of ".bin" files now. and an odd index.txt + index.bin
worthy cape
I haven't updated it with the index findings yet.
@dull laurel A bundle has a fixed header and several blocks of Oodle-compressed data. Each block (apart from possibly the last) decompresses into 256k of data.
The files in a bundle are laid out end-to-end after decompression with no visible delimiter. The size and order of files in a bundle are determined by the index.
The text index contains bundle names and the filenames that are in that bundle, in order.
dull laurel
fun stuff. I wish they would share the format to make life easier for 3rd party people
worthy cape
The binary index is a multi-layered file that has a list of bundles with their uncompressed size, a list of all files telling what bundle index they have and their offset/size in the uncompressed blob. It also has an encoded directory tree which we're still figuring out how to re-assemble.
This should be roughly the state of things. It is currently possible to extract a file from a bundle given its location in the text index by some indirection.
For the standalone client, there's also .bin files but they're held inside a GGPK for patching convenience.
dull laurel
just checking the folder. there is a "regular" version for each /Data file and a "dat64" version of it. I wonder if you stitch those bundles together you get the actual file.
worthy cape
dat/dat64 existed in the past, they're the same but dat64 has wider types.
There's now also datl/datl64, which have UTF-32 text I believe.
timber path
@worthy cape did you manage to complete the oozLib? Would it be possible to get a decompression method using the following method signature? byte[] Decompress(byte[] compressedContent, int decompressedSize) (c#-syntax)
worthy cape
In order to extract a file you need to figure out where in the uncompressed blob it resides and expand the blocks it covers and slice it out.
@timber path The one I've got now exports a function with the same signature as the Oodle lib.
Your signature would require allocation and deallocation for the return value... how does C# interop deal with that?
dull laurel
and they dont use an open source library for their files?
worthy cape
@dull laurel It uses RAD's Oodle library, which is very commercial and good.
dull laurel
ah
timber path
Your signature would require allocation and deallocation for the return value... how does C# interop deal with that?
@worthy cape Not sure... I could also pass a buffer pointer & int size, which is the current signature I believe?
dull laurel
but oodle only for compression or also for splitting the bundles?
timber path
I just wanted to avoid Marshalling stuff in c#, it makes the code look ugly and complex
worthy cape
@dull laurel The compression is only for the bundle blocks.
Ah, I seem to not have pushed my recent commits.
timber path
@worthy cape do you have a compiled dll? And under what license is it available?
I'd like to start some work on a c# lib that can read the index, compose a list of bundles & their respective files, and have the option to retrieve the content of a given file
worthy cape
This just thinly wraps Kraken_Decompress, I moved my PoE-specific stuff to a separate DLL that can load either this or the proper Oodle DLL for decompression, which deals with bundles and whatnot.
dull laurel
and the official one is not even available for just decompression for free? 😦
worthy cape
It only ships with licensed games, like say, randomly, Warframe or Doom Eternal.
PoE static-links, so no go there.
dull laurel
haha, how mean
timber path
This just thinly wraps
Kraken_Decompress, I moved my PoE-specific stuff to a separate DLL that can load either this or the proper Oodle DLL for decompression, which deals with bundles and whatnot.
@worthy cape sounds good. All I needed was the decompress method in a dll. The entire bundle stuff & index reading I've already coded in a c# library
worthy cape
There's bits in the source code where there's a SAFE_SPACE of 64 bytes allocated after destination buffers, but I don't know if that's vestigal or actually scribbled on by the decompressor.
timber path
From a glance at the kraken.cpp it looked like it is used during the decompression process, where the space is used to write outside of the actual uncompressed size, as part of the decompression process
worthy cape
I implemented some canaries for some particular bundles but it didn't clobber it, but that's not exhaustive.
dull laurel
mh, so you could copy the dll from the other games? i mean warframe is f2p too. you just cant bundle it or upload it to github, right?=
worthy cape
Perfectly "fine" for offline tooldev, royal pain in the backside for anything distributable.
timber path
Even for offline tooldev it's fishy... You've used illigal/incorrectly licensed software to obtain a given result, which you then distribute... doesn't sound right
vapid pulsar
I have a way to call the decompression function inside of the PathofExile.exe but im not sure if it breaks the tos
worthy cape
@timber path Yeah, lots of sketchy things all around.
dull laurel
but it sounds like fun, like a fun project to do until the league starts 😄
golden bane
@vapid pulsar It sure does. Refer to section 7b of the ToS
- Restrictions: Under no circumstances, without the prior written approval of Grinding Gear Games, may you:
b. Modify or adapt (including through third parties and third-party tools) the game client or its data, other than in the normal course of PoE gameplay as permitted in accordance with the Licence.
warm zodiac
I can understand when u doing big project at work but cmon its poe
simple ravine
@worthy cape the inner index bin thing, is it necessary to deduce which file is which or?
I am kind of hinging on the smarter people here to understand the bundle things in order to get the GGPK/DAT etc reader in C# working 😇
worthy cape
Chuan claims that all you need is index.txt and the outer section of index.bin - https://discordapp.com/channels/174993814845521922/175290321695932416/754309358736637982
simple ravine
ah yes, hmm
worthy cape
I'm not sure if the file part of index.bin is ordered or if you need to sort by bundle+decompressed_offset, didn't really look at that much.
The inner blob seems to be more about having a compact encoding for all full paths.
Checking some things around that, but need to run out for groceries soon 🙂
worthy cape
There's exactly as many bundles and files in the index as there are in index.txt, and each file path is part of exactly one bundle.
timber path
@worthy cape is there a proper template for that yet? Probably it's faster to compase file names using that inner bundle, instead of trying to figure it out using the inded.txt ?
worthy cape
I don't know if the inner bundle contains any bundle names/indices or if you still need the text file for that.
timber path
It's taking my c# app quite some time to just create the bundle->files structure
worthy cape
Like 1s here with hot files.
timber path
How? Mine takes over 70 seconds to compose the list
The slow part seems to be where I have to lookup the file name, using the bundle index to first lookup at which index the bundle starts in the TXT
worthy cape
That is, from index.txt grouping paths per bundle, and from index.bin grouping file entry to bundle entry.
I should also build some LUTs for bundle name to bundle index and path to bundle name, I guess.
timber path
How do you efficiently group the txt? without knowing if it's a bundle or file?
worthy cape
When reading the binary bundle info, I populate a lookup table of valid bundle names and check each line of the file against those.
timber path
Or knowing how many files are in a specific bundle...
worthy cape
std::string line;
while (std::getline(iss, line)) {
if (line.size() && line.back() == '\r') {
line.resize(line.size() - 1);
}
//fprintf(stderr, "%s", hex_dump(16, reinterpret_cast<uint8_t const*>(line.data()), line.size()).c_str());
auto bundle_I = bundle_index_from_name.find(line);
if (bundle_I != bundle_index_from_name.end()) {
last_bundle = bundled_filenames.insert({ line, {} }).first;
++bundle_line_count;
} else {
if (!filename_occurence.insert(line).second) {
fprintf(stderr, "Duplicate file \"%s\" in bundle \"%s\"\n", last_bundle->first.c_str(), line.c_str());
}
last_bundle->second.push_back(line);
++file_line_count;
}
}
(disregard the check in the else clause, it was just to sanity check that there were no files that lived in multiple bundles)
timber path
That's similar to how I build it. I first read all bundle names from the index, then loop over the files and lookup the bundle index in the txt and then offset it by the file index
worthy cape
Takes maybe 13s in Debug.
timber path
70 sec in debug for me
// Read the file info
int fileCount = reader.ReadInt32();
int lastBundleIndex = -1;
AssetBundle currentBundle = null;
int bundleIndexInTxt = -1;
for(int i = 0; i < fileCount; i++)
{
// Read unknown values.
reader.ReadInt32();
reader.ReadInt32();
// Read known values.
int bundleIndex = reader.ReadInt32();
int offset = reader.ReadInt32();
int size = reader.ReadInt32();
// Find the file name
if(lastBundleIndex != bundleIndex)
{
currentBundle = Bundles[bundleIndex];
string bundleName = currentBundle.Name.Substring(0, currentBundle.Name.IndexOf(AssetBundle.FileExtension));
bundleIndexInTxt = indexTxt.FindIndex(x => bundleName == x);
lastBundleIndex = bundleIndex;
}
string name = indexTxt[bundleIndexInTxt + currentBundle.Files.Count];
currentBundle.Files.Add(new AssetFile(currentBundle, name, offset, size));
}
Is that FindIndex a linear search?
timber path
Maybe it's more efficient to build the bundle->file-names list first
It's the default c# implementation of FindIndex... not sure how that works
worthy cape
Yeah, it is.
timber path
The List<T> is searched forward starting at the first element and ending at the last element.
Yup, looks like it's just linear
Binary Search would be faster?
worthy cape
Binary search, ordered dict or hashmap all are sublinear, but none preserve the insertion order.
So if you have some index order you want to keep, store the index with your data.
Maybe a dense List with bundle information and a Dict from name to list index.
cursive sigil
is that just a binaryreader too?
timber path
@cursive sigil yes, I'm using a binary reader to read the bundle.bin's decompressed content.
Figured a faster way to map file names to their respective bundle names.
cursive sigil
the reader/stream for doing many small reads is also working against you; though 70s still feels crazy
timber path
When looping the bundle list, I add all bundle names to a HashSet; next I loop over the entire indedTxt and check each line to see if it's a bundle name (using a Contains on the HashSet), and fill up a dict (bundle - list<filenames>) in that way; this makes the entire thing sub 1 second
The HashSet is what did the trick here
cursive sigil
not super familiar with dict, but you could also skip the hashset and just use your dict as the 'set'?
worthy cape
Indeed, populate it with empty bundles as you sweep the binary first.
simple ravine
@timber path try with .NET 5 if you're not
I'm about to set on a journey to read the bundles. How are you decoding them atm?
simple ravine
hmm... so entry 46 and onwards in the _.index.bin in Bundles2 does not start with 80 0C as the 45 previous.
they start with CC 06
worthy cape
Still valid, I believe.
Low nibble of first byte is Ch for Kraken, bit 6 means "uncompressed"
bit 7 means "restart decoder"
Second byte selects decoder type for the low 7 bits and whether to use checksums for the high bit.
simple ravine
https://encode.su/threads/3068-Leviathan-Updated-Kraken-Decoder#post60677 according to this there's a CC 07
worthy cape
The packed size of the entry 46 is 262146 bytes, which is 256k + 2-byte header, incidentally.
simple ravine
I am glad there are some real smart people here 🙂
worthy cape
@simple ravine It makes sense that the tail of the bundle isn't compressed, as that's where the inner bundle might start, which already is compressed.
simple ravine
There're a couple of mermaid ones after
mortal bone
This is one of those things I would like to get into at some point haha
simple ravine
I'm gonna watch some 'Lovecraft Country' while doing this lol
Leviathan, Kraken, Mermaid
Makes sense too
worthy cape
Haven't made any leeway on the string shenanigans, slow Sunday.
mortal bone
ah rad tools?
simple ravine
indeed
mortal bone
those are some smart fucking dudes
simple ravine
probably yeah, nonetheless quite annoying new shenanigans to use zao's terminology
mortal bone
did they bundle all of the .dats? I should probably just scroll up a few days haha
simple ravine
they bundled everything into bundles, yeah
mortal bone
oh shit
mortal bone
that was in a x.y.2 release?! lmfao
worthy cape
@mortal bone https://zao.github.io/poe-doc/bundles.html
mortal bone
thanks man
simple ravine
zao, whitefang, chuahsing et al have made some good progress
i've been asking stupid questions
worthy cape
Lacks half the stuff we've found, but is a good starter.
They intentionally deployed the tech patch before the league end so the worst bugs would shake out before Heist.
It also gave us a whole week of hair-tearing time before Heist launch 😄
simple ravine
also considering this is the first version of this, it's not unlikely there will be some changes
mortal bone
Nice, I wish I was of some use here. I need to read up on my reverse engineering data streams
simple ravine
unless they thoroughly tested this 
mortal bone
It has been a very long time since I have messed with any of this
simple ravine
yeah, same here. i opened some of the files and looked at the patterns, went total ADD
mortal bone
this will help the steam launcher significantly at least
simple ravine
funny story...
steam ditched ggpk
standalone packs these bins inside the ggpk still
mortal bone
oh shit
why maintain the two different styles? wouldn't it just be easier to throw them into a single /data/ folder or some shit
I guess you are guaranteed a contiguous block of disk space with one file
simple ravine
I had just ditched the abstraction of IFile and IDirectory that had an implementation for both virtual files in GGPK and regular files for stuff like my Dat reader. I guess I will take them back 😂
worthy cape
Still have a solid patcher with the tree of digessts, just over a 15th of the files or so.
simple ravine
I think what he's asking is why ggpk for standalone, and non-ggpk for steam
mortal bone
Yeah, and I think I answered my own question
simple ravine
or did I misunderstand the steam part?
worthy cape
Steam handles the file juggling and patching and there small disk files are a necessity.
mortal bone
Yeah, I know the reason for steam, but the standalone being different is interesting
worthy cape
For Standalone it'd have to do a lot of disk inventory to figure out what to patch.
worthy cape
While if things are still in the GGPK, they have their existing SHA256 stuff.
simple ravine
potentially something they'd do at a later point in time
worthy cape
Steam knows what version the game is supposed to be and "should" keep that integrity.
mortal bone
Ah, yeah, that makes sense
worthy cape
I haven't looked at the existing Standalone patching protocol, but I think it's something like exchanging hashes across levels of the tree to find out what parts to alter.
simple ravine
They mentioned something about not having to send half a million of hashes across the wire, indeed
ah no, half a million requests.
vapid pulsar
The standalone probably keept the ggpk so they can memmap the whole thing. Then they can access and file in it with a memory offset and don't have to worry about the Io stuff
simple ravine
Wonder if there are names for the "files" inside of the _.index.bin
worthy cape
What do you mean by "files" there?
It'd be interesting to see if there's an order or relation between the ones holding {bundle-idx, offset, size} and the mysterious string soup.
simple ravine
considering there are a bunch of entries in there
they're nameless though
an index of the index? 😄
odd sail
does anyone here care to explain at what level the bundles start on the standalone client?
I assume I still open the Content.ggpk and then parse the two records of GGPK type
velvet fog
odd sail
yeah, I read that
I guess I'm not understanding
so they are contained in the top-level ggpk?
golden bane
I recommend you to open the GGPK in PyPoE and you'll see
obtuse citrus
I've been looking at the file names linked from the index bin
I'm not entirely sure if the logic works for everything but it seems to builds a composite string from the paths
\x00\x00\x00\x00 appears to be some sort of separator that denotes specific sections
and non-zero values at the first byte to indicate which position to subsititute into a new composite string
Small function that might help to make it a bit more readable. ```python
def make_paths(raw):
paths = []
offset = 0
rawlen = len(raw)-4
while offset <= rawlen:
index = struct.unpack_from('<I', raw, offset=offset)[0]
offset +=4
if index == 0:
paths.append(None)
continue
end_offset = raw.find(b'\x00', offset)
string = raw[offset:end_offset]
offset = end_offset+1
paths.append((index, string))
#pprint.pprint(paths)
return paths```
One of the examples I've looked at for instance:
\x01\x00\x00\x00 Art/Microtransactions/pet/Dragon/models/\x00
\x01\x00\x00\x00 dragonlight.a\x00
\x01\x00\x00\x00 landing\x00
\x01\x00\x00\x00 rig.a\x00
\x00\x00\x00\x00
\x04\x00\x00\x00 md\x00
\x04\x00\x00\x00 st\x00
\x03\x00\x00\x00 02.amd\x00
\x03\x00\x00\x00 02.ast\x00
\x03\x00\x00\x00 .amd\x00
\x03\x00\x00\x00 .ast\x00
\x02\x00\x00\x00 md\x00
\x02\x00\x00\x00 st\x00```I believe it generates Art/Microtransactions/pet/Dragon/models/ Art/Microtransactions/pet/Dragon/models/dragonlight.a Art/Microtransactions/pet/Dragon/models/landing Art/Microtransactions/pet/Dragon/models/rig.a in the first step
and then Art/Microtransactions/pet/Dragon/models/dragonlight.amd Art/Microtransactions/pet/Dragon/models/dragonlight.ast Art/Microtransactions/pet/Dragon/models/landing.amd Art/Microtransactions/pet/Dragon/models/landing.ast Art/Microtransactions/pet/Dragon/models/landing02.amd Art/Microtransactions/pet/Dragon/models/landing02.ast Art/Microtransactions/pet/Dragon/models/rig.amd Art/Microtransactions/pet/Dragon/models/rig.ast
via the second part after the separator
worthy cape
File count for bundle 17692 ("_Tiny_2") mismatch - binary (73182) and text (121138)
File count for bundle 17693 ("Metadata/EnvironmentSettings") mismatch - binary (231) and text (197)
File count for bundle 17728 ("_Tiny_1") mismatch - binary (139094) and text (107683)
File count for bundle 17729 ("_Tiny_4") mismatch - binary (36076) and text (19565)
Do I have something wrong in my code?
The total number of bundles and files match, so I haven't misidentified any lines.
timber path
After my c# port and optimizing the bundle-tree composing I did encounter a similar issue. So there seems to be a mismatch or some bundle name accidentally also matches a file name within a bundle
velvet fog
but the sum up is correct
timber path
The binary sum up is correct, but not our own file count from the txt
Especially when looking at zao's code posted earlier, which I've also adopted in c#. it loops over the index txt line by line, once it encounters a bundle it'll make a list for that bundle and adds all subsequent lines to said list until it encounters a file name that matches another bundle name; at which point it starts a new list. So, when a bundle and file name accidentally match, it'll start a new list, thinking it's a new bundle, while it actually isn't
worthy cape
Bundle count in index binary: 17730
Bundle count in index text: 17730
File count in index binary: 515438
File count in index text: 515438
It would have to misidentify both ways then, which I don't quite see how it would happen.
timber path
With my theory I'd expect more bundles in the txt though... 
worthy cape
All lines in the text file are unique.
timber path
For tiny_2 your txt ends at 121138...try to print the next index to see what file name is there?
worthy cape
Time to get started on the workday, but will not be able to avoid this during the day 😛
simple ravine
hmm, I am getting 'file too big'
when I chop my index up and try to run it thru ooz
velvet fog
should we report the file count for bundle mismatch issue to them?
simple ravine
that'd be hilarious
worthy cape
@simple ravine For standalone ooz.exe you need to prepend a 64-bit binary header with the uncompressed size - https://github.com/zao/ooz/blob/master/kraken.cpp#L4454-L4460
simple ravine
thank you!
i shouldn't code past midnight... i was real good at this in my 20s. i have to accept i am not anymore.
worthy cape
I'm sure you can UML all the things tho 😉
vapid pulsar
What are the currently missing pieces with the bundle stuff?
If we are stuck on something ill run through the disassembly in the exe tonight
simple ravine
haha, im pretty bad with UML too. thankfully, i dodged the TOGAF hype too
but im fairly good with distributed systems, know a thing or two about PACELC/CAP theorem etc
wait, so this is just 1 "file"
worthy cape
I wonder about the unk[2] in the file info blocks, are they hashes akin to the murmur2s of the past?
@simple ravine The inner bundle? Yeah.
simple ravine
all night, yesterday, i thought this was several files due to the chunks
and because they had 80 0c etc at several places
im confused, because i see only 1 uncompressed size
worthy cape
It's a big slab of data sliced by the table that comes before the bundle.
There unk1 is the offset, unk4 is the size. The slice contents are a mix of u32 and zero-terminated narrow strings: https://gist.github.com/zao/50eabba0812ad32fb9020136e41e8ad8
See slightly above for Omega's try at interpreting them, they are compact representation of paths that share some prefix.
simple ravine
yeah I was poking at the uncompressed last night, and saw the second number was always fairly small
this part is a bit confusing though
velvet fog
Omega had decoded above
worthy cape
See 06:11 and onward.
simple ravine
will do, thanks
worthy cape
Shower thoughts are the best thoughts...
simple ravine
I will gain some of those quite soon too
worthy cape
What if the inner bundle is the authoritative set of filenames?
There's one field in the table preceeding it that often is the size of the path lump but sometimes not, I wonder if that might be a linked list, much like FREE chunks used to be.
simple ravine
My naive thought would be that it might be something similar to the Dat format's data chunk, which could be either strings or, a list of references etc
worthy cape
I've got a LMDB file with the inner lumps sliced out, but not sure if that's of much use when you can do it yourself.
simple ravine
Dat format is
<header>
<table, fixed length per row>
<0xBBbbBBbbBBbbBBbb = separator>
<bunch of data, strings, lists of offsets etc>
so _.index.bin only have 1 uncompressed size, so I assume I should treat it as 1 contiguous set of data. Not sure why there are chunks tho, perhaps part of how the algorithm works?
worthy cape
A bundle itself knows nothing about the data inside, it just knows about blocks.
Without external metadata (like the one contained inside the index), all a bundle holds is compressed data with block-oriented random access.
With external information about the extents of files, the bundle can be selectively decompressed to reveal the storage for a file.
The index itself is a bundle of a single big "well known" "file", which contains an inner bundle again containing one big "well known" "file".
simple ravine
what confuses me, when using WhiteFang's modification passing the -e (explode) option, it produces 54 x 256 chunks. 54 being the length of that chunk array
worthy cape
IIRC, it dates back to when the list in the bundle was thought to be entries, not blocks.
Haven't looked at the source lately, but I'm guessing you're getting each 256k block in a separate file?
Ah no, that was -p.
simple ravine
yeah, the index file, in decompressed state, is it supposed to be read as 1 big 13MB file, or separate ones?
I assume the former
worthy cape
For any non-index bundle, -p -e should get you individual files based on the index.
The index payload is a single file of "magical index" type.
simple ravine
Heh, that renders my assumptions and code I made last night pretty useless in its current form. https://github.com/andreandersen/PoeSharp/blob/new/src/PoeSharp.Filetypes/Bundle/EncodedIndexBundle.cs
worthy cape
It's kind of like the filesystem metadata on a disk, it's not a file file, it's just implied.
simple ravine
the chunks are fairly "useless" in its encoded state with other words, as you decode the whole thing
with other words, I take the whole thing, prepend the uncompressed size from the header, and decompress it as 1 thing, rather than prepending 256*1024 on each of those 54 chunks
worthy cape
@simple ravine Not sure if you can bulk decode multiple blocks, existing code iterates over them in turn and concatenates them in memory.
Compress something with the standalone compressor and see if there's headers every 256k I guess.
timber path
what confuses me, when using WhiteFang's modification passing the -e (explode) option, it produces 54 x 256 chunks. 54 being the length of that chunk array
@simple ravine Those 54 files are the decompressed chunks. They all have to be appended back together (without any kind of separator) to create a big byte-array which is an X number of files together and can be separated again using the file info contained inside the index.bin
When using my custom ooz with -p -e arguments, you also get the chunks appended together in a single file; You don't really need the -e actually, it was there as an early experiment
simple ravine
I think I need to wake up still, didn't have much sleep, so I feel I am using up a lot of ya'lls good patience with my questions which I could probably answer if I spent a bit more time with reading the code, but it's a bit confusing, still. I'll tinker a bit more after shower & breakfast, and see if it "clicks".
Thanks Whitefang & zao
timber path
My latest custom-ooz commit also adds the option to add the index.bin & index.txt to the args; this'll decompress a given bundle.bin into its respective files rather than its chunks of data
simple ravine
yes, did a git pull this morning and saw this, very nice addition
timber path
Please careful using that... Some bundles contain like 13k files!
It'll be an explosion on your disk
simple ravine
the reason for me tinkering with this, is that I have a library PoeSharp
and I'd like to at least attempt to re-implement the ooz code in c#
timber path
I have a working c# bundle reader dll
I got all except the ooz part in it. So it reads the index and creates a proper bundle-file-tree and you can retrieve one specific file
simple ravine
It's probably a bit too much to take on, but I thought "hey, at least I'll re-learn some c++ as I go along, which can't be bad." as it's been 16-17 years since I had a basic c++ course, and since it's always good to understand some c++, why not.
And this is a pretty good thing to tinker with, as I enjoy PoE stuff, and not actively programming in my new role at work, I want to keep my programming skills a bit fresh, so a hobby-project like this is nice
and I'm a sucker for trying to squeeze out the most performance out of .net 🙂
yea i know 😂
timber path
The oodle decompression thing in c++ is really complex to convert to c# ... Would be nice, but Zao's oozlib.dll works fine for my usecase though
simple ravine
well, I "just" need the decompression part... I'll give it a day or so, to see how much head-banging there will be
I'm fairly sure I'll have to give up, but we'll see
brb
flint fractal
Yo so quick question guys, what's the most optimal way to check if a boss is dead from client.txt? are there specific voice lines for every boss in the game?
violet path
not every, but some
flint fractal
hmm, i guess detecting boss kills is very unreliable then, atleast if u want to automate it?
simple ravine
@timber path or @worthy cape
I prpended the file with the uncompressed size as mentioned, which made the file look like this in the beginning:
7A 9F D5 00 8C 0C 00 87
also tried with a long
7A 9F D5 00 00 00 00 00 8C 0C 00 87 EB 88 57 90
however, that didn't work
ooz is saying
ooz _.index.bin.prepped _.index.bin.decoded
dec->src_used=34801
offset=262144
outbytes=-1
_.index.bin.prepped: decompress error
are you using the dll or the builtin version @simple ravine ?
simple ravine
using ooz
I prepped it using my own code, so ooz would "understand" it
prepending the uncompressed size before the 8C 0C
vapid pulsar
the builtin decompressor in ooz has an issue with newer versions of oodle compressed files. You need to use the dll. if you put the oo2core_7_win64.dll in the same directory as ooz and add --dll it will work.
worthy cape
I've only decompressed particular files (index, inner index, StatDescriptions, some minimap).
Haven't done any exhaustive tests.
simple ravine
this is the _.index.bin
so what I did was pretty much take out the poe-added-header, take the meat and prepend the uncompressed size and resave it
worthy cape
I decompress chunk by chunk tho, as I mentioned I'm not sure if stock ooz understands the idea of concatenated chunks.
vapid pulsar
I think it does. each chunk has what is internally called a quantum header. I have had stock ooz work on some multi chunk files and fail on other. If i use the dll flag it works evey time for me
simple ravine
How did you determine the right uncompressed size to prepend each chunk with?
vapid pulsar
i know the poe clients runs the entire multi chunk blob through as a whole, not in chunks
simple ravine
public static void PrepareIndexForDecompression(
Stream source, Stream destination)
{
ValidateSourceStream(source);
ValidateDestinationStream(destination);
// Read the header stuff
var hdr = source.Read<IndexBinHead>();
// Read the chunk sizes
var sizes = source.Read<uint>((int)hdr.EntryCount);
// Write uncompressed size as bytes
// to destination stream
var prepSize = BitConverter.GetBytes(
(long)hdr.UncompressedSize);
destination.Write(prepSize);
// Write the rest of the file
source.CopyTo(destination);
}
basically did this... did not work
worthy cape
@simple ravine There's a field in the bundle that happens to hold the value 0x40000 (256k). All chunks but possibly the last are 256k uncompressed, the final one is the remainder.
simple ravine
Just funny that the compressed sizes are varying from 33kB and 183 kB, and the tail being 253 and 256kB
timber path
i know the poe clients runs the entire multi chunk blob through as a whole, not in chunks
@vapid pulsar How do you know? Did you inspect or decompile the exe?
vapid pulsar
Yeah, it even has some helpful debug logging inside of it
timber path
@simple ravine I'm not sure if vanilla ooz support decompression of multiple chunks though... Maybe you have to feed it chunk-by-chunk? - Atleast that's what I'm doing:
// Read and decompress the entry bytes
byte[] decompressedContent = new byte[uncompressedSize];
byte[] decompressionBuffer = new byte[MaxChunkSize + SafeSpace];
int lastEntry = entryCount - 1;
int offset = 0;
for(int i = 0; i < entryCount; i++)
{
byte[] compressedContent = reader.ReadBytes(entrySizes[i]);
int decompressedSize = (i == lastEntry) ? (uncompressedSize - (lastEntry * MaxChunkSize)) : MaxChunkSize;
LibOoz.Ooz_Decompress(compressedContent, compressedContent.Length, decompressionBuffer, decompressedSize);
Array.Copy(decompressionBuffer, 0, decompressedContent, offset, decompressedSize);
offset += decompressedSize;
}
return decompressedContent;
thanks, I'll try that
I just wanted to first off get a chunk that ooz can decompress, so I can haev 1 compressed and 1 decompressed size
timber path
@vapid pulsar interesting... Maybe that (entire chunk-blob) only works when using the official oodle DLL and not ooz.
simple ravine
which I can use for live unit testing while writing the decompression
also a mental process i guess, wrapping my head around this
vapid pulsar
yeah that does seem to be the case. if you use ooz with --dllit works fine
timber path
Ah ic. I avoided using the dll since I have no license for it anyway... So If I relied on it, i'd have to get a license for any apps I build on/with it.
vapid pulsar
the dll is inside the poe exe and you can call it but i still need to get an official answer if that will be too far as you have to actually spawn the exe and hook it.
broken cloud
@flint fractal no way to automate, very few bosses emit lines in Client.txt... maybe set up a hotkey to whisper yourself a "boss killed" signal?
@ everyone else good luck all reverse engineers! 
flint fractal
@broken cloud That's actually an amazing idea, cuz then u can determine boss on area i assume, so if "boss killed" was written and last area entered was "Shaper's Realm" = +1 Shaper kill
Would be harder on guardians tho, i would need 4 specific binds for those i guess
broken cloud
Not quite 🙂 Uber elder fight is also The Shaper's Realm
Shaper is one of the bosses that has a "fight complete" line though... The Shaper: Irrelevant!
worthy cape
»why are our server logs full of people whispering "honk" to themselves?»
flint fractal
Ah right, hmm, imma have to come up with something smart for that i guess, maybe some overlay stuff where i can select what boss i killed or something
simple ravine
whispering yourself, to then react on a client.txt log feels like taking strange detour
broken cloud
You have to whisper yourself to make sure it gets through... just using local chat is unreliable, you get muted after a while (I assume it's some sort of spam detection)
flint fractal
I haven't rly done any POE app work before this small project, but would it be hard to make an overlay work together with fx a simple WPF c# app?, Fx if i hold down a button it opens up an overlay with boss icons, where i can click one and that would be the boss i killed, and it would send that information to the WPF app running, (dno if u can do the overlay in same app)
worthy cape
Primary restriction for anything on top of PoE is that you need to run in a windowed mode.
Exclusive fullscreen means injecting overlays, which is way more sketchy.
flint fractal
I only ever play in borderless, which would accept overlays aswell right?, but do you guys have any recommendations on overlay tools that can work/interact with C# applications or something?
worthy cape
You can probably look at how existing tools do things. Some hotkey to pop somewhat transparent or shaped windows on top of PoE without decorations, and dismissing themselves when you click through to the game.
Awakened PoE Trade, PoE Overlay, etc.
Not sure if any of them run WPF.
flint fractal
I'll have a look, thanks alot!
violet path
check for when the player's ladder exp goes up equal to an amount a boss gives, then credit the boss kill?
simple ravine
@broken cloud If you have an application that is triggering that in the first place, why go through those hoops in the first place, is what I meant.
key-pressed -> app -> send whisper via poe -> log -> app -> do_work
why not just
key-pressed -> app -> do_work
broken cloud
Ah I see, laziness in my case (already monitoring Client.txt for lots of stuff, let's do this there too, one less thing to think about)
simple ravine
ooz .\idxprep\_.index.0.bin .\idxprep\_.index.0.dec
outbytes=262144
.\idxprep\_.index.0.bin: 34809 => 262144 (0.00 seconds, 510.90 MB/s)
boom!
ooz .\idxprep\_.index.53.bin .\idxprep\_.index.53.dec
dec->src_used=106363
offset=106361
outbytes=-1
.\idxprep\_.index.53.bin: decompress error
😦
ooz .\idxprep\_.index.52.bin .\idxprep\_.index.52.dec
outbytes=262144
.\idxprep\_.index.52.bin: 262154 => 262144 (0.00 seconds, 1884.57 MB/s)
looks like I am doing something janky on the last chunk
ah nvm figured it out
off-by-1 errors
timber path
Not sure if any of them run WPF.
@worthy cape I can speak for PoE Overlay Community Fork, it's no WPF; we use Electron for the overlaying part. Might save @flint fractal some work trying to figure that out 😄 (The original PoE Overlay now uses Overwolf, I'd suggest avoiding that, since there was a big fuzz about it, that's why the community fork exists in the first place)
flint fractal
Ah, thanks alot 🙂 Can you make Electron interact with fx a WPF app in any way? (I don't know alot about what electron is)
timber path
Any particular reason to stick to WPF, which has been dead for 5 years now?
simple ravine
It's not dead.
They re-introduced it in .NET Core 3.1
They're also building out WinUI 3 for it
timber path
Ah, ic... They've put new life into the project then. I didn't know that. Probably I'm out-dated then on its state
Maybe under a new name then... because the WPF blog is dead since 2015 ... https://docs.microsoft.com/en-us/archive/blogs/wpf/
Still wondering what @flint fractal 's reason is to want to stick to WPF anyway...
simple ravine
Some update:
https://www.youtube.com/watch?v=EMQNt_rqNBY
There's also MAUI coming out, but that's targeted for .NET 6 IIRC
flint fractal
Simple answer is i'm studying programming and our main language is C#, I've already done a few MVC projects, the only thing i haven't touched at all is WPF, so i wanted to do something with it.
timber path
Ah ic... Electron is just the framework that's being used, the language for UI is mainly html with javascript, where the code is written in Typescript; but you can use c# for it too, if I'm correct...
worthy cape
Use what you know even if it's not the "best" - just look at @simple ravine insisting on C# 😛
simple ravine
flint fractal
I've heard about typescript recently in C# aswell, if i'm not mistaken
I'll look into it @timber path Thanks alot!
simple ravine
I am just stubborn to show that a more high-level language like C# can actually perform as well as a C/C++ application if made the right way. 🙂
Trade-offs etc
golden bane
In workloads that are IO bound, many languages become more viable
simple ravine
Not only that. The .NET IL JIT can optimize the execution in a way that might make your application faster, actually. Of course, that requires you to write that code carefully and with thought, and not use bloated LINQ queries in hot paths etc.
golden bane
Yeah, knowledge of how your code maps to actual instructions helps (and is often pretty interesting in itself)
barren ingot
how do you guys know that the game content hasn't been encrypted (in addition to being compressed) ?
timber path
@barren ingot because after decompressing for example a stat_descriptions.txt, it became human-readable text, as it did prior to 3.11.2 🙂
golden bane
Because we're able to make sense of it just fine, albeit slowly
barren ingot
i guess the likelihood of optionally encrypting some and not all files is low
worthy cape
My biggest concern about data is whether textures are plain DDS still or have scrambled blocks for better compression.
timber path
There's also no real benefit in encrypting the files; I assume the trade-off against loading performance is too high then
golden bane
More general answer: the decompressed content is not uniformly randomly distributed
barren ingot
i was looking into steam update pipeline documentation for inspiration, on the assumption that in order for GGG to improve steam client releases they may have internally changed their mechanism to be more like steam
timber path
They've exploded the GGPK for steam, and that's as far as I know more in line with squizing out the best, or atleast better, performance out of steam updates
timber path
A layer of encryption would only be beneficial to avoid people poking around in files, but slows down the overall performance to load such huge amounts of assets into the game, since you'd have to both decrypt and decompress them
mortal bone
@worthy cape they did mention that they get better quality textures, but that might just mean they are using lossless compression
timber path
@barren ingot it's unclear from your article if steam chops it up in chunks for you, and compresses + encrypts it... or if that's the sugested way to go, but it's still up to you to actually do that.
barren ingot
it looks like it's left up to the game developer to decide that and they just provide guidelines for if you do want to do that, how would you best do it to not interfere with the diffing mechanism to result in more efficient patches
mortal bone
Internally they still use the ggpk
It appears they changed the patcher for the steam version. I would be curious to see if copying the exploded ggpk to the standalone client would still work
timber path
@barren ingot yeah, it looks like that indeed... I think GGG went half-way here, but using their own bundling (is it their own though? maybe the bundle.bin is a common known way of shipping files?) system & oodle's compression system.
mortal bone
Bundle comes from oodles compression
timber path
@mortal bone You'd have to pack it into a GGPK though, but I think it would work, except for the exes, since those are steam-specific
mortal bone
There is additional metadata outside to find the chunks within the bundle from what I have gathered
timber path
I mean... would there be a reason for it not to work?
mortal bone
I was curious to see if you didn't have to repack it
timber path
This metadata outside, is the _.index.bin ; which is also present in the standalone ggpk
barren ingot
what do you mean "changed the patcher for the steam version"? I believe steam patches work by the game developer promoting a new build, and the patcher compares files and looks for chunked deltas like rsync to fetch and apply. I believe GGG changed the structure of their large GGPK file so that there are more "file boundaries" so that when a delta is calculated, it's contiguous and localized
timber path
Oh, in such way... I can just paste the standalone exe in the steam folder & run it to see the effects
barren ingot
i.e, making one small change doesn't cause a large section of change in the ggpk file
mortal bone
@barren ingot yeah, I know
They exploded the ggpk for the steam version and kept it in the ggpk for the standalone version
worthy cape
@mortal bone Is there prior art in other software that uses Oodle that has similar "bundle" files, or is it a GGG invention?
barren ingot
so the steam client has a bunch of bundle files in the game directory, versus the standalone client has one large ggpk?
worthy cape
Indeedily.
Same payloads, but standalone has it in an extra GGPK layer for their patching logic.
barren ingot
got it
mortal bone
Oodles does have a texture compressor that is separate from the data compressor
barren ingot
well that's nice, radgametools offers oodle SDK for free if you ask. is that what you gys did?
"Oodle is easy to use, cross platform, and super fast. Don't take our word for it, email us for a free evaluation SDK. "
worthy cape
@mortal bone That's the one I'm scared of, as it has the permuting BC7Prep option.
@barren ingot That line kind of assumes that you're a company or prospective customer, probably comes with a nice NDA.
We've used an open source reimplementation and/or loose DLL files from other games.
timber path
"evaluation" sdk... probably limited in time
Maybe someone could actually ask them? instead of assuming?
barren ingot
is the algorithm academically sourced so that it could be reimplemented accurately? if not, i find it unlikely for an open source reinterpretation of a closed source algorithm to be successful
those types of projects tend to be competition to closed source options, as opposed to a 1:1 compatibility
simple ravine
why do you say that? we've actually used one, and it worked.
barren ingot
why do i say what? i think what i wrote is generally accurate
worthy cape
@barren ingot A lot of the inner workings are described on ryg's and cbloom's blogs over the last bunch of years.
barren ingot
ah ok that makes sense then
worthy cape
rANS, tANS, arithmetic coding and all the other buzzwordy stuff.
mortal bone
I would assume they used bc7prep if they are talking about texture quality improving. Let me see if I can't find other games that use the system. I am only familiar with the tools because every major indie game dev studio talks about them
worthy cape
I would assume that they were getting good results with RDO alone.
BC7Prep has a runtime cost, which might not slot in that well into the renderer.
Feels like something you'd use only if you were really tight on space to eke out the extra few percent from the compressor.
Time will tell when we get around to grabbing actual DDS files.
mortal bone
They also just might be using a less lossy compression and getting better results
nova crown
anyone reversed the new .bin file format?
worthy cape
@nova crown Mostly.
nova crown
is the info public anywhere?
golden bane
This chat, scroll up a bit
worthy cape
The last bunch of days in this chat since the tech patch, also partially written documentation.
I ought to fix that tonight.
nova crown
aight imma look in here then, thanks
worthy cape
@obtuse citrus The two-phase algorithm for templates and generation seems to hold. Of note is that there is sometimes multiple zero-delimited sections which resets the templates.
So you can have:
0
1 "foo"
1 "bar"
0
2 "baz"
0
1 "qux"
0
1 "x"
1 "xx"
0
Generating "foobarbaz", quxx, quxxx.
The first template section can also be empty, then the second section generate as-is without a prefix.
obtuse citrus
Good to know
I still need a better way to decompress the stuff then using a command line too. It's a bit of a pain
I think it would be ideal to just use poe's static linked functions assuming that works fine
mortal bone
might be a bit of a grey area haha
worthy cape
Yeah, what's your thoughts on distribution, as you distribute stuff.
obtuse citrus
That's why I'd use the static functions instead of distributing anything
Can't push 3rd party dlls to other people
mortal bone
Yeah, you would need access to poe.exe and you would not distribute it
obtuse citrus
the other thing is GPL'd which is annoying
worthy cape
Of note is that the GPL reluctantly allows integration even via a plugin interface if the interface is simple enough like "feed input into plugin main, get output back".
mortal bone
technically, not modifying, but I could see the argument for adapting. Although, you are technically using the code for its intended use lol
velvet fog
macOS patch in 7 hours
obtuse citrus
if you want to argue
streaming is also displaying derivative works of the game
as is the wiki
or any guide
anything really
mortal bone
To be fair, those are explicitly permitted.
golden bane
For PoB, we'll most likely opt to call ooz as an external command, not distribute it ourselves and keep the exporter MIT
mortal bone
If you really didn't want to get your hands dirty you would have to reimplement the decompressor under whatever license you choose...
worthy cape
I've adapted it both into a shared library with a standalone decompress function, and also as a child process daemon that takes and returns framed data.
Simple enough composition doesn't spread the license.
golden bane
I can live with violating the spirit of the GPL, but technically complying with it
Fuck the GPL
worthy cape
grave wren
MIT gud
golden bane
Quote me on that whenever you like lol
worthy cape
$ wc -l inner_probe.txt
2455176 inner_probe.txt
$ sort inner_probe.txt | uniq | wc -l
515438
This is odd, there's a fair number of duplicate generated strings in the inner blob.
obtuse citrus
the license crap has been generally been argued far too much and it's complicated mess depending on each countries laws and can very from court to court whether something works or not
velvet fog
there are many odd things in this patch
obtuse citrus
Not just in terms GPL, but also in regards to terms of service, derivate works and so on
But in any case if GGG wants to go to court against people providing value for their game, for free, I don't see that turning out well for them :P
worthy cape
Ooh, maybe some slices are repeated in the some array beforehand.
obtuse citrus
It's a good way to kill their good rep within a short time
@worthy cape wasn't that just kraken decompress?
aren't there other formats from oodle in use?
golden bane
Kraken, Mermaid and Leviathan
nova crown
I am looking at some of the code posted in here and I see talks about uncompressed sizes, but some files have readable strings in them
are they not compressed or what's up?
timber path
@nova crown they are compressed. Probably the compression algorithm doesn't, or can't, fully or properly compress certain things... Not sure though, just wild-guessing here... You know it's compressed by the 8C06, which is the octet used to identify a Kraken compression
Also there is a different purpose for each of the 3 used compressions, some might be better/faster for specific purposes, but might have a down-side of not compressing everything properly, or have a special way of compressing or something... I dunno
mortal bone
DDS files will be compressed, but they may be (most likely are) using a different compression algorithm for them because those are texture files.
timber path
Compression is not meant to make things "unreadable" 😄
simple ravine
I was wondering if it would be a good idea to have a collaborative space that is not filled with chat, where N+1 people can contribute with findings and documentation?
For the sole purpose of these new shenanigans
nova crown
most of the time things become unreadable when talking compression
simple ravine
Just a simple shared scratchpad
nova crown
that's why seeing the whole DDS magic seemed weird
worthy cape
@nova crown I see you found that part as well. Some compression schemes embed literals in the stream as early on often the most efficient way to describe the data is the data itself.
There's also some chunks that explicitly use the "uncompressed" decoder, typically things that are already compressed like the tail of the index.
grave wren
is there anything like that publicly available @simple ravine
mortal bone
We can always start a tools discord lol
worthy cape
@obtuse citrus While it's called "Kraken" in the interface it speaks several of the underlying decompressors with the same framing.
simple ravine
@grave wren hmm, good question. there are pobably quite a few, question which one is the simplest to use while satisfying the basic needs of gathering this for now
grave wren
for starters i know nothing besides a more private chat that'd work
simple ravine
I'm thinking quite low-ambition right now, just like a shared space to throw some stuff in
worthy cape
I've got my docs/notes, but those are very non-collaborative 😄
simple ravine
if google hadn't discontinued google wave, that'd been fantastic
mortal bone
We really need something with a chat + wiki type thing
simple ravine
i concur
mortal bone
*** Microsoft Teams Enters Chat ***
simple ravine
yeah, the overhead of O365 license enters the room
mortal bone
*** Microsoft Teams Leaves Chat ***
mortal bone
something like slack or rocket chat?
simple ravine
something that isn't chat, more like wiki/mardown-esque googledocs-esque?
tiny cargo
Once a upon time people used forums. Separable by sub forums, different topic, searchable, while posting is ez and fast, even sticky topics
simple ravine
thinking out loud
nova crown
isn't that just what trello/github projects are for?
simple ravine
the main problem I want to solve is finding the information, where having a working document space, with this chat as a separate flowing thing
obtuse citrus
@worthy cape ah that makes sense, I guess I'll try that out. Decompressing via system commands is annoying and slow XD
grave wren
sounds like some kind of notebooky thing
simple ravine
yeah
How about Notion.so?
not sure if it's real time, but I think that would be a great thing
very rich and free
mortal bone
I just started a Github Projects, and it is just a kanban board with no chat features lol
oak estuary
GitHub has a wiki too
simple ravine
Free version, everyone is admin and it's not like an open space though
Where people who are not collaborating to 'read'
so that's the downside of that tool, but other than that, great tool
nova crown
@mortal bone there aren't many (free), if any, that over the whole set of features, wiki type notes + chat
and why would they try to do it, unless you are talking corporate
golden bane
@mortal bone For PoBC, we went back to using a google drive spreadsheet after trying out GitHub Projects for a league or so lol
mortal bone
might just be easier to start a different discord and use google docs lol
golden bane
Looks like a "chat, wiki, unlimited use" pick two kind of situation
simple ravine
What I wanted to also achieve (bonus) was for newcoming people who come here and ask, as some have done, where we could point them to this place as a 'read' thing
mortal bone
a github wiki isn't a bad idea in an organization
simple ravine
let's try that
grave wren
gdocs sounds ew
simple ravine
what's the worst that could happen? it doesn't work, we lost 30 minutes
simple ravine
cool
add me 🙂
zao, whitefang5, chuan etc probably would be willing to share their notes, I hope 🙂
my username is andreandersen
mortal bone
sent, sorry, haha
simple ravine
I think you need to add privileges to the ggpk.discussion
mortal bone
Ah, I can make a team and it has a discussion section
simple ravine
I guess we'll just try different things out, and we'll find the best way soon
mortal bone
if I add you to the team on the organization I can default privilege to admin on that repo
simple ravine
yeah, let's see if I or zao can invite others, in case that is desired
can't seem to invite new people
I guess that's up to you to deal with Emmitt 🙂
mortal bone
Ah, I have to make you an admin in the parent org, one sec
barren ingot
i've used https://gitter.im/ with other os projects before
simple ravine
gitter is a chat on top of github iirc
mortal bone
ye
simple ravine
what was desired though, was a collaborative space that isn't mixed with an ongoing discussion
like a separate "meeting notes" type of thing
best way I could explain it from my pov
mortal bone
Alright, I have made both of you owners in the parent org, so you should be able to manage the team. I wish there was another level like Team Owner
simple ravine
yeah, could use Azure DevOps, but that only has 5 basic licenses, unless you bring your Visual Studio License (MSDN)
but that stuff has extensive permissions etc, based on Azure AD
barren ingot
are you a not-so-secret msft evangelist 😄
mortal bone
I like jira workflows for tasks a lot better than azure devops
simple ravine
hmm interesting
mortal bone
It could just be my organization blows at setting that all up
Like, when I start a task I want the parent story to start and when I close my task I was the successor notified that it was closed
simple ravine
yeah a lot has to do with your personal experience, which a lot of the time hinging on how well things are set up etc
ah, yeah i think some won't want it that way (maybe most? not sure), but I think you can configure it... I recall that being possible in TFS etc
but indeed, JIRA is very extensive that way
mortal bone
Yeah, I will have to look into it. I just got admin access, so I can start improving our workflows
barren ingot
you guys must have very interesting backgrounds. i don't personally know too many people willing to try and reverse engineer game data storage. i can udnerstand why poedb would invest in doing this as its directly correlated to their business but everyone else i'm not sure 😄
worthy cape
@velvet fog I was wrong before about duplicate strings in the inner bundle. I had used unk4 instead of unk3 as the length. With unk1 as base and unk3 as size, the ranges completely cover the inner bundle data once, and there's 515438 paths, all unique.
(this is the exact amount of file entries there are in the index)
velvet fog
That's great, I will check this tomorrow, time to sleep
simple ravine
@velvet fog, @timber path - would you be interested in an invite to PoE-Tool-Dev workspace which Emmitt set up for us to share the findings, and dump your thoughts in there, as well?
If so, I'll get you invited, so you can share whatever you think is worthwhile
worthy cape
I'm gonna go conjure up some dinner, not sure what to try after that.
mortal bone
@barren ingot I haven't done much with the GGPK, but this type of stuff is how I got into programming. I like to know how things work, and I started out making bots for various games. A lot of the time it was just packet replication, but then I wanted to make headless clients and want not. I got into exe disassembly and such after that. I now just work a normal desk job and do that kind of stuff on the side (although have put it off for some time now)
simple ravine
A normal desk job, but programming or no?
It kind of felt like you were talking about developer stuff at work recently
mortal bone
Yeah, I am a mid-level dev at an insurance company
I mainly work with Guidewire products, so a lot of it is Gosu
simple ravine
googles Gosu and Guidewire
mortal bone
I pretty much have to do all my fulfilling dev outside of work sadly
shitty proprietary language that compiles into java byte code
simple ravine
My condolences
mortal bone
basically, C# + TypeScript in the JVM
worthy cape
@barren ingot I'm a HPC sysadmin with a focus on compiling/debugging scientific software, and a hobby interest of the icky parts of systems/game programming, language interop and other madness.
PoE file formats are a fun and frustrating puzzle, so many of them.
mortal bone
Yeah, I would like to do something different, but I can't pass up the health benefits, and it pays well. It just doesn't scratch the itch when it comes to complexity
Also, I work with some of the worst developers ever lol
worthy cape
Step 1) Fix corona. Step 2) Senpai GGG notice me. Step 3) ???
mortal bone
Ayyy, if I lived in AUS or NZ I would have tried for a job lol
They had a web dev position a while back that I would have liked to try for
simple ravine
I draw boxes and lines. Mostly with fluffy clouds.
mortal bone
If you ever have a position open that is remote let me know lol
simple ravine
I sold my share in my company, went back to working at Sogeti (Capgemini sister company)
grave wren
i'd throw jira out if i could
simple ravine
@barren ingot but yeah, other than drawing diagrams, I've been doing programming since a little kid... pretty ok with it. Now they call me 'chief cloud architect'
grave wren
it's basically not at all compatible with our entire workflow
mortal bone
The one thing I like about Azure DevOps (especially with git) is it is compatible with an ass load of workflows
I wish test plans and what not could be a bit more integrated with automated testing outside of Visual Studio Test Explorer though
simple ravine
reason why i was unhappy with jira (except it was a pretty old one, i guess... probably 10+ years ago i used it), was that the hosting we had with it was sluggish, outright slow
they're deprecating the automated testing, @mortal bone
mortal bone
ah, ok
simple ravine
guess they rather integrate with leading vendors, and then buy them up 😂
@mortal bone but mid-level quite soon then, or has time flown quick? I recall you being and finishing school since I joined
I guess it's like 4 years I've been here, but timeline is a bit fuzzy
mortal bone
Yeah, I just hit 4 years of professional experience, but I have been out of school for 3 years.
I am just good at my job haha ¯_(ツ)_/¯
I am hoping to hit senior at the end of this year
simple ravine
hobby-programming before right?
mortal bone
Oh yeah, I have been programming for ~12 years haha
simple ravine
yeah then I think you'd probably good for a senior title
worthy cape
Señor Emmitt
simple ravine
be aware though, and i don't say this to be negative towards any technology or product, but i've seen people who have been quite niched, and grown in ranks, feel a bit "lost" when changing jobs to something more general (like a "normal" dev job, or what have you)
but seeing what you've done here and stuff, i think you'll be fine for sure
seen 'senior solution architects' at our company being totally lost when stepping out of the sharepoint bubble
timber path
Hobby programming years count for next to nothing... Found out the hard way
grave wren
I still find the senior title funny considering every company redefines it
guess it's just denoting that the person isnt totally lost without lead
simple ravine
@timber path you mean in job-advancement purposes, or actual skill progression?
timber path
With 10+ years of hobby programming, a degree in application development and computer games technology it still took me 100+ job applications and 1 year of searching to actually land a job lol
grave wren
my experience was the polar opposite with a msc in media compsci
4 applications 4 offers 2.5 years ago
simple ravine
junior: "I am invincible, I can do anything" 0 - 3 or 4 years
mid-level: "What am I even doing?" 4 or 5 years - 7 years
senior: "I might be getting the hang of it, but... yeah"
years are my personal opinion, generic
timber path
In the end it all worked out though, I've been a happy game developer now for 4,5 years in a small game dev studio that is growing every year. 🙂
simple ravine
@timber path difficult in your geographic region in general?
grave wren
did you specifically check for game dev or general dev?
timber path
Difficult geographic region... No game dev schools, a handful of "indie" companies... I moved and applied abroad in many different countries
The 100+ applications was specific game dev companies only. If I just wanted a job, I could've taken a boring software development job and make some dusty accountant software or stock manging thingy lol
simple ravine
I'm fortunate to have worked on some interesting things (not a game dev).
Designing geographically distributed systems that needs to be really secure and performant etc
timber path
I have years of experience in C++ coding an old-school mmorpg... That's where I learned almost all my coding and work ethics tbh
But that's just "hobby experience" which counts for 0 years in the industry lol
mortal bone
@simple ravine yeah, I am aware. I switched companies almost a year ago, and I was able to pick up a new language/software without too many issues. I have actually been improving our code and performance haha I am currently leading our upgrade project
simple ravine
nice!
mortal bone
I mean, hobby experience counts for a bit when you can show something
timber path
Ofc there are other intersting things to work on; Between my two degrees I've worked for almost 2y in a medical software company that has software running in almost all hospitals in Belgium... So atleast I contributed something useful to society 😄
simple ravine
yeah... there are many 'expert beginners', who don't challenge themselves and just tinker with the same thing for a long time
but then again you have Sharepoint developers who spend years managing lists and workspaces, professionally
so I think it all depends on the individual at the end of the day
grave wren
We got an sap GUI who only did forms for 20+y
simple ravine
interestingly, software development industry is kind of special, on good or bad... most people who are really good are people who have done programming as a hobby since they were young, and it's a passion
timber path
Oh, I like challenges, so I've mainly brought a lot of innoative ideas to the old mmorpg source code... Even rewrote the entire networking layer of the server(s) & client... So it's not like I'm tinkering around doing some minor additions or changes to some source code 🙂
simple ravine
I think for hobbyists, mainly WhiteFang. I think it's more "normal" to be a little challenged at a professional gig, but that's not always the case either of course
grave wren
It doesn't irk me as special re: good people also do it as a hobby
timber path
@simple ravine yeah, I agree. I've seen devs get out of school without it being their hobby who just sucked at what they do, but managed to graduate by lifting on the backs of the passionate hobbiests
I think the hobbiests do learn a lot more prior to their jobs, but that exp isn't regarded at all. It's just the piece of paper & work experience that gets seen
simple ravine
I remember me handing over a pre-study I did for a client, this is 10+ years ago now.. and he was fairly fresh out of school. He had literally no idea what was waiting for him out in the real world.
He had learnt about facades and factories, but had no idea about real stuff. It was a web-based thing, and had no industry-preparedness. It's unfortunate that schools are so academic and not preparing them for proper stuff.
Game-devs, I think is a bit different though. You have better use for binary trees, tries, how a linked list is made, and how to write a bubblesort. lol
timber path
Depends on the game-dev jobs... As an engine programmer, sure...
But as a more gameplay-dev, you'll never touch or consider that low-level stuff at all.
You're just playing in the sandbox called Unreal Engine or Unity most of the time.
timber path
game-dev is a wide term to describe you're a programmer in some game company
simple ravine
at least need to know the performance quirks and really keep an eye on the stack, and heap etc, knowing the tricks
timber path
But you could be working on the Engine, or UI, or gameplay, or scripting, ...
simple ravine
TIL
timber path
at least need to know the performance quirks and really keep an eye on the stack, and heap etc, knowing the tricks
@simple ravine You should, but most devs, mostly those fresh from school, know nothing about those things.
simple ravine
I guess SIMD/AVX/SSE tricks are mainly used in the engine then?
timber path
My colleagues often are baffled when I arrive with some weird bit-shifting operations juggling pointers around to cast an int to an enum in c#
The engine, especially Unreal & Unity, often takes care of memory management for you.
grave wren
That sounds not very maintainable 
simple ravine
I like to tinker with low-level performance code, but not to the extent of bit-shifting. I don't know what it is, but I can't for the life of me grasp it. Perhaps I just need to sit down and actually try to understand it exclusively for a night or so.
timber path
Simple things like a flags-enum contains method that uses bit-operators baffles them
public static bool Contains(this RoadType types, RoadType type)
{
return ((types & type) != 0);
}
This kind of things gets them confused already... Due to the bit-operator & lol
When I arrive with things like:
public static int NumberOfTypes(this RoadType types)
{
int i = (int)types;
i = i - ((i >> 1) & 0x55555555);
i = (i & 0x33333333) + ((i >> 2) & 0x33333333);
return (((i + (i >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24;
}
their brain explodes
simple ravine
I mainly do things like
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe T To<T>(this Span<byte> buf)
{
fixed (byte* b = &buf[0])
{
return Unsafe.ReadUnaligned<T>(b);
}
}
yeah, I try to refrain from that kind of code that is difficult to read and maintain, unless it's utterly necessary.
timber path
Ah, pointer stuff :p lovely!
public static void Set<TEnum>(this SaveObject saveObject, string key, TEnum value)
where TEnum : unmanaged, Enum
{
unsafe
{
saveObject.Set(key, *(int*)&value);
}
}
public static TEnum Get<TEnum>(this SaveObject saveObject, string key, TEnum defaultValue)
where TEnum : unmanaged, Enum
{
unsafe
{
int enumValue = saveObject.Get(key, *(int*)&defaultValue);
return *(TEnum*)&enumValue;
}
}
that looks a little strange, but alright i can maybe dig it
timber path
How else to convert an enum to int without allocations? Maybe you can teach me a thing or two? :p
The underlying SaveObject here can only store primitives, so no custom types at all.
simple ravine
you mean intermediary allocation for the sake of casting?
timber path
Yeah, any kind of temp. allocation just to cast the enum to int, or visa versa. This code is an allocation-less generic cast of any enum to its int value, or visa versa. (Using generics in c#)
mortal bone
how the hell did you where : Enum?
timber path
c# 7.3 allows this
mortal bone
ah shit
barren ingot
you guys are doing all the right things for an IC role in software engineering. 90% of companies with software as a product will mean you will stagnate in your tech stack. changing tech stacks or learning new ones tend to be orthogonal to adding any business value, unless you're SO outdated you can't hire anyone proficient in what you're using 😄
simple ravine
barren ingot
i transitioned a few teams a few companise ago off of C# because in the boston area the talent was so limited and hiring new college grads to work in c# was basically limiting our pool to those not with the times, at least in the web services space
simple ravine
.method public hidebysig
instance int32 M (
valuetype SomeEnum e
) cil managed
{
// Method begins at RVA 0x2050
// Code size 2 (0x2)
.maxstack 8
IL_0000: ldarg.1
IL_0001: ret
} // end of method C::M
I have some vague memory of them taking care of the boxing/unboxing. Not sure if this is the case, or if the test is too simple so it gets too optimized.
but yeah, your extension method isn't pretty good 🙂
timber path
The method in your screenshot works if you have 1 enum... doesn't work anymore when using Generics to cover any enum though.
simple ravine
True, but before you had the issue of an box and unbox. That's not the case anymore, I believe
IL_0000: ldarga.s e
IL_0002: conv.u
IL_0003: ldind.i4
IL_0004: ret
When using normal casting? maybe... it's just that c# 7.3's generic T:Enum can't be cast at all, because System.Enum doesn't inherit from int.
simple ravine
is what you get from
public unsafe int M<T>(T e) where T : unmanaged, Enum
{
return *(int*)&e;
}
so is there a "core" unpacking library where the work is being coordinated on? I was only aware of PyPoE by Omega
worthy cape
Separate initiatives mostly.
barren ingot
or since the format has changed, is this the first opportunity to actually have a core library for reuse by different solutions like wiki, poedb, etc
timber path
There's just some common knowledge on the file structure & content, but each one of us is making, or made, their own implementation/adaptation for their own tools I believe
simple ravine
meanwhile non-generic will give you
IL_0000: ldarg.1
IL_0001: ret
which is the equivalent of a mov and ret
but then again, if you have many enums..
barren ingot
well i hope the first to arrive at something working can release a library for others to consume like how omega did 😄
timber path
which is the equivalent of a
movandret
@simple ravine which is what I aimed for with the method, but it doesn't actually do that? Strange though... The main thing is, it has to use generics
mortal bone
Um, Omega_K2 has PyPoe, zao has one in Rust iirc, thezensei has one in C#, and chuanhsing has his for poedb
simple ravine
yeah, I can't inspect the JitASM on generics
so can't know what the exact instruction would be
worthy cape
Started on a decent C API for bundle and file access to make it easier to FFI, but I quickly moved to exploring in standalone projects, as making an interface is hard when you don't know what you're supposed to expose.
timber path
@simple ravine the pointer juggling I'm doing, would suggest a simple move & return, atleast, that's what I aimed for, using my C++ pointer knowledge.
simple ravine
conv.u Convert to unsigned native int, pushing native int on stack.
ldind.i4 Loads the int32 value at address addr onto the stack as an int32.
mortal bone
@worthy cape ah, ok. That makes sense haha. It would be nice to have a decent c api for it since pretty much anyone can just invoke in whatever language
then people could just make a wrapper in whatever their language of choice was
timber path
I agree with @worthy cape ; I have my own c# library to read stuff in the bundles, mainly for the purpose of extracting very specific files so I can transform them into json's I need for PoE Overlay Community Fork... So a generic library that does a lot of different things is difficult to make, especially not knowing what the user of the lib actually wants or needs
worthy cape
But yes, this is a great opportunity to align forces.
barren ingot
so all these other tools you guys know, don't have a dependency on pypoe?
simple ravine
brather1ng is providing JSON data out of the PoE stuff, mainly using PyPoE IIRC. Very good initiative as well. https://github.com/brather1ng/RePoE
mortal bone
I think the file format/reading can be defined pretty explicitly then you can choose to do with it what you want.
barren ingot
(historically speaking, pre: 3.12 changes)
golden bane
Also PoB has its own GGPK ecosystem
worthy cape
High-level consumers tended to use pypoe offline or repoe for the baked Data.
simple ravine
I have mine, PoeSharp, but I haven't worked on it a lot. Perhaps some day it'll be more friendly for people to use
barren ingot
so POB doesn't depend on PyPoE in any way?
worthy cape
All my junk was custom, got like three different GGPK loaders and writers.
golden bane
No
simple ravine
Haha, same as zao. I tend to dive into rabbit holes, and then throw it away, and start over.
simple ravine
worthy cape
eyes the 144 GiB sqlite3 db with deduplicated PoE files
simple ravine
my condolences, zao
mortal bone
time to brush up on the C bois
I wish I would have done that with the skill tree data
just apply a "patch" to major json changes...
grave wren
you could put it on a micro sd and frame it zao!
worthy cape
It took like three months to build, so somewhat precious 😄
barren ingot
in terms of library development, i think
- a library/util that extracts ggpk files
- a library/util that handles POE specific bundling info and eases decompression into individual data files
- a reader of POE data files into language-specific classes for further consumption into other things
would be nice
mortal bone
eww who wants these on disk 😛
golden bane
Write it in Haxe so you can target most languages "natively" 
simple ravine
https://github.com/andreandersen/PoeSharp/tree/new/src/PoeSharp.Filetypes @barren ingot .. but again, very sporadic work on it the past couple of years.
barren ingot
pfft storage is cheap just buy one or two ssds and use it as your working directory
golden bane
Or REST is also always an option I guess
barren ingot
wat
simple ravine
good storage isn't super cheap though, like real good NVMe that actually lasts
worthy cape
I almost considered buying a 14TB disk for the GGPK project.
barren ingot
you want to expose poe game data for a web api?
mortal bone
hell yeah
worthy cape
I had a ggpkserve that exposed a GGPK over HTTP. It was horrible.
barren ingot
you EU guys with spoiled internet
worthy cape
wget --mirror took a literal eternity.
barren ingot
over here in america i pay like 170 usd for gigabit fiber
golden bane
Not over the internet, you would spawn a local server
barren ingot
imagine living in SE and having the option since 1992 or whatever
i was always jealous of bredsbandbolaget or however it's spleled.
those were the guys you wanted to be your best friend on IRC
worthy cape
Depending greatly on if you're in a Good city or not, only recently has fiber started to permeate the countryside.
Still a fair bit of cable and DSL 🤮
mortal bone
I live in a small town, but we have fiber. 10/10 would recommend
barren ingot
i was practically setting you up for the past ten min to show off like that
it's about time
grave wren
dont worry way too much effort to type it off
mortal bone
grave wren
maybe if someone wrote a bot or something
worthy cape
So what things are people looking at now?
barren ingot
i like how emmitt did not expose their ip
mortal bone
I am looking at work and discord
barren ingot
fast learner
simple ravine
right now, IL code and enum conversions. ADD. Rabbit hole.
couldn't let it go. lol.
grave wren
adding a backend to @regal mural finally :>
worthy cape
I'm able to generate the full file list from the index tail, but not quite sure what to do with it or what the meaning of the other field in the generation list is for.
simple ravine
the idea was for me to stupidly trying to implement the kraken stuff in c#
barren ingot
wow you're going to implement a whole compression mechanism
worthy cape
I guess I should diff the file lists against the index and see if they match?
simple ravine
ant, just the decompression. going to ignore the compression
timber path
So what things are people looking at now?
@worthy cape I'm looking into using my own lib for the PoE Overlay Community Fork so it's prepped when Heist launches. I've also just pushed the lib project (that's part of a bigger sln though), it's a simple c# lib that can read an unpacked content.ggpk (or rather, the steam Bundles2 folder) and compose a bundle/file tree and allowed retrieval of individual file contents. (Thx again for the libooz.dll !)
(For anyone interested in it: https://github.com/WhiteFang5/PoE-Asset-Updater/tree/master/PoEAssetReader )
simple ravine
i probably will hit a brick wall within the first 60 minutes, and give up 😄
barren ingot
🧱 pls stop now
mortal bone
also, I haven't been able to find any other game packing these files in this way.
Outside of the PS5 dev kit including Oodle Textures by default not many people seem to publish much about it
barren ingot
i think it's an interesting move by GGG to make such a change that would knowingly brick the data source for community tools without guidance on how to transition into reading the new format
these tools exist because there's no official equivalent
simple ravine
I think that's quite far down their priority list. Players come up top 🙂
barren ingot
could there be an official equivalent coming? 😮
i don't have a clear understanding into how many of these tools are even used by everyday players. i was talking to another friend and there was some stat thrown about hte % of people that make it to maps. presumably people that don't make it to maps are unlikely to use any of these tools
worthy cape
Imagine Harvest without poedb/craftofexile.
simple ravine
maybe we'd regain some of the feeling of mystery and exploration back 😄
worthy cape
Ah yes, manually updated and inaccurate wiki pages. 😄
barren ingot
i'm just talking about this stuff to gauge the "business" decision to obsolete a bunch of third party tools. i.e, after 3.12 gets launched is reddit going to be up in pitchforks about pob, poedb etc not being accurate anymore
golden bane
@mortal bone FIFA 18/19 uses Oodle
mortal bone
Ah, thanks
grave wren
i dont think thats much of a concern
compared to complaints about slow loading in steam
worthy cape
"we would've told them the specs if anyone had just asked"
barren ingot
lol
worthy cape
(not an actual quote)
barren ingot
the moment i saw the patch notes i was lke uh oh poedb tears
worthy cape
This has been in the air since that post on reddit months ago.
mortal bone
I would be super interested if GGG was using BC7Prep, because from what I have seen, you don't really need it. You already gain a lot from switching to Oodle Texture + Oodle <Data Compression>
too be fair, this should also significantly reduce load times for various assets Oodle is fucking nuts when it comes to decompression
barren ingot
what is "watching river" ?
worthy cape
@barren ingot The API that drives trade sites is the "river", an endless stream of JSON chunks with the contents of public stash tabs.
Third party tools and trade sites can consume the river by fetching changes from some change-ID and iterating forward in time, trying to keep up.
barren ingot
oh ok, i just call that the public stash endpoint -_-
simple ravine
river, because it's not worthy of the name firehose
but cool enough to have a nickname 🙂
I think 'firehose' in modern api times originates from Twitter's Firehose, which is like super exclusive... one reason is that most people won't have the capacity of taking on the firehose
mortal bone
This is interesting
The frostbite engine does something similar to what GGG has done. Except they have bundles in bundles which contain files
simple ravine
curious to understand why you would have bundle in bundle
simple ravine
so they're easier to skip if not needed to load, perhaps
mortal bone
Less disk reads
simple ravine
but on the surface, it feels a little like zipping zip files together
which i find mindboggling
worthy cape
Mandatory xzibit reference.
mortal bone
They also might be able to run compression on the bundles? I didn't dig into it too much. I doubt it would gain much, but maybe something
Consoles be weird man
So, should we create tools discord? We can have various channels for pob/pypoe/ggpk parsing etc?
I know the various tools have their own discords, but more for developers rather than the public
simple ravine
I'm torn.
I'd of course join. The reason why I am torn is if it warrants multiple channels
worthy cape
I come here for the news channels and this one ^_^
simple ravine
Same. I am not very active in the other channels. I think it's a lot of noise in there, but I might not have given it enough chance. I admin the community started by Lifting as well.
'The Forbidden Trove' also has a tooldev chat, I just recently discovered. Quite a few there as well. So spread out.
actually theirs is fairly silent... and i have really no idea what their community is made of tbh
mortal bone
The problem with this is we don't really have a way to breakout general from a specific task (albeit not a problem 90% of the time)
grave wren
seems like just adding a new temp channel for ggpk stuff might be the call
simple ravine
@mortal bone u think this server is not the right place to add more tooldev channels, or?
mortal bone
We would have to ask <@&174997701513969665>. It might be useful to have a channel group for tool-dev that has general and then any specific task that needs to be worked on like ggpk parsing, etc. rather than sticking us under "Other"
mortal bone
Um, I think it is, but I don't manage the server haha
hexed wasp
Whatchu want?
simple ravine
cat pics
grave wren
banana bread
simple ravine
brb dinner.
mortal bone
I can talk in the other channel, so we don't clutter this
serene belfry
timber path
Is it possible to get the chinese poe downloaded without vpn?
worthy cape
I've now verified that the set of file paths generated from the inner tail is identical to the set of file paths in the text index.
mortal bone
#tooldev-breakout bois
we can use that channel for any discussions that need a more team effort
barren ingot
after reading the chat log for the past few days is it accurate to say that
- ggpk has been able to be broken into bundle files using the metadata in the index file
and that - the current open problem is decompressing bundle files into the familiar DAT format that existing tools have been written against?
worthy cape
The web of bundle files contains most of the files that were previously loose in the GGPK.
While we have understood how to decompress bundle chunks, the mapping of filenames to file entries is not quite robust and understood.
barren ingot
oh really? i thought the index file contained the names and offsets of bundles
gentle yacht
I thought the breaking down of ggpk was only done in standalone?
obtuse citrus
I wouldn't mind a general tool dev discord, this is almost too public facing
mortal bone
haha I know what you mean
worthy cape
@barren ingot There is a bit of uncertainty whether the text index is broken or if there's some misunderstanding of the binary index.
barren ingot
never know when a ggg person will just hop in and say pls use this link to download a library in the language of your choice to read game data 😉
obtuse citrus
I wish but I don't see that happening
barren ingot
ok so the current issue is assigning human readable names to files?
obtuse citrus
Some people here that that work at GGG are very helpful though when they're allowed to be though
mortal bone
I would be interested to see how OpenArl is going to update POB lol
barren ingot
what if you skip assigning names to files--
does decompressing the files result in usable things? i.e, can you decompress and interpret what the file is based off its contents instead of caring about its name
mortal bone
that is a lot of insider info that can be made public real quick
worthy cape
@barren ingot I haven't tried, I think that someone looked at what seemed to be dat files. Probably one of the next steps.
obtuse citrus
You kinda need to know what dat file you are dealing with in regards to the specifications
Similar with other stuff like the .ot/.otc stuff can't necessearily infer from the contents
worthy cape
For 3.11.2, straight-up checksums can help map out the filenames, but it helps less for future or changed content.
barren ingot
well, what if you just attempted to read every file using dat.py from yor PyPoE -- what's the likelihood that they changed the actual DAT format. from the patch notes and from my understanding of more efficient steam patching, they changed the organization but not necessarily the smaller units
pseudo ocean
how long has this channel been here and why am i suddenly seeing it now?