hello deer
#tooldev-general
barren ingot
worthy cape
Approximately forever, and it changed groups I guess.
obtuse citrus
it needs to know what file it is to parse it properly
when the patch hits probably several dozen .dats are changed
so it isn't really helpful to try them at random
barren ingot
everything is speculation until it's tried π
obtuse citrus
well then go ahead and try
it's better to properly understand how file names associated then banging your head against a wall
mortal bone
The definitions for the file formats changes every league (sometimes between normal patches). You can't just try all of them because you don't have the definition lol
barren ingot
so they're not self descriptive?
worthy cape
@obtuse citrus I linked my implementation of the tail generation scheme in #tooldev-breakout for reference btw.
obtuse citrus
neat, I still need to implement it properly, just happy that I figured it out
I think I'll try our your dll prior though
the the commandline stuff is seriously stupid
worthy cape
Realm down... plz2gief patch π
velvet fog
macOS patch I guess
worthy cape
My Standalone just got a 25 and 128 MiB patch.
velvet fog
Bundles2/_.index.txt removed
worthy cape
Ooh?
simple ravine
nice.
mortal bone
Was index.txt just a debug file that got left in? Haha
worthy cape
I guess I'll definitely try some reverse lookups tomorrow, see if I can find any files with a digest that has had a single owning file in the past, to lock down some paths.
I prefer to see it as a friendly push in the right direction π
There we have Steam patch too.
Massive 32.1 MiB π
Indeed:
Removed file β Bundles2/_.index.txt (37.07 MiB)
velvet fog
I don't know how inner_bundle connect to _index.txt, the name resolver issue
worthy cape
Have you looked at the unk[2] array at all? Is there any connection between a file entry and the path specs by the inner bundle?
velvet fog
not yet
cursive sigil
wait seriously, they removed the index.txt too?!
they seriously don't want us to read it...
worthy cape
@cursive sigil Upside of things, there's no longer any bundle mismatch between text and binary π
cursive sigil
now we can have tons of files and not a clue to what anything is π
velvet fog
ya :p
vapid pulsar
Does the data in unk[2]look random? or is there some pattern?
simple ravine
fyi, @vapid pulsar
#tooldev-breakout was created, so we can have a separate channel for special discussions like this topic.
also created github organisation + project/wiki. let me know if you want an invite, to contribute with findings and documentation.
obtuse citrus
yeah feel free to send an invite :)
mortal bone
@obtuse citrus sent you an invite.
obtuse citrus
thanks
vapid pulsar
Ill take an invite too π https://github.com/eingin
velvet fog
invite me https://github.com/Chuanhsing π
mortal bone
Sent :) for both
We can always change the name of the team/repo. @worthy cape added his bundle notes to the repo wiki already
exotic niche
Could i get an invite too ? Don't have much experience but did some digging and googling. https://github.com/Paliak
mortal bone
Invite sent. Here is a link to the wiki for those that don't want to contribute but want to see: https://github.com/PoE-Tool-Dev/ggpk.discussion/wiki/
exotic niche
Thanks
timber path
I'm also interested to contribute...
mortal bone
sent π
lean spindle
^ I will help out where I can as well https://github.com/John-Paul-R
mortal bone
Also, there is a discussion tab in the team that is private to the members of those teams if there is things that are a tad more private
velvet fog
got zero progress this morning :<
velvet fog
_.index.txt removed, should we remove the related part in wiki?
timber path
@velvet fog probably a good idea. I see no reason or need to keep that (now out dated) info in it.
lean spindle
Quick question:
I am attempting to read the _.index.bin file using the information on the wiki.
It mentions that "The binary index file is a compressed bundle..."
Does this refer to the entire file, or just the end? How does one go about decompressing this if the former?
vapid pulsar
@lean spindle It follows the same format as other bundles as described on that same page.
https://github.com/PoE-Tool-Dev/ggpk.discussion/wiki/Bundle-scheme#bundle-file-format
The compression is done by a 3rd party library called Oodle.
lean spindle
Ah! it was right there in front of me. Sorry about that, and thank you for the help!
vapid pulsar
As far as decompression goes it's a bit tricky as the Oodle sdk is not open source so we are currently relying on someones re-implementation of it. or the dll that is part of some other games like warframe
https://github.com/rarten/ooz
lean spindle
Understood, appreciate the info o7
dull laurel
didn't you want to use the other chatroom for that? π
worthy cape
@dull laurel Please try to keep unrelated discussions out of the breakout room, it got quite hard finding relevant information here after a few watercooler chats.
dull laurel
you got me confused then @worthy cape. i thought the new channel was to discuss the new file format?
worthy cape
Yes, but rants about C#'s capabilities will invite noise.
dull laurel
I see
timber path
meh, why does C# have to be so ugly. no map / reduce / whatever on array types? π¦
@dull laurel use Linq?
dull laurel
@dull laurel use Linq?
@timber path yeah, that worked, after figuring out why Sum() doesn't like ulong/uint64. now i know why you use int/long in your code π
timber path
Yup, that is exactly my reasoning for not using unsigned, because Linq doesn't play nice with those primitives π¦
simple ravine
Good thing about LINQ, is that they're basically just a bunch of Extension methods, and you can write your own extensions very easily π
dull laurel
@timber path is there any way to debug this libooz? I tried to copy what you did, but the decompress only gives me 00s
worthy cape
Ensure that it succeeded - if it fails it returns -1, otherwise the amount decoded.
timber path
@dull laurel Not really... Atleast not in an easy way. I.e. it's just an exposed version of the ooz tool's kraken_decompress method. You could export you input bytes, prepended with an int64 which contains the decompressed size to a file, then feed that into the ooz tool...
dull laurel
@worthy cape ah, that helps. returns -1 all the time. just no verbosity on the error
worthy cape
dull laurel
set the output buffer size to 256 instead of 256kb. π
golden bane
Newer versions of the Oodle .dll return 0 instead of -1 on failure btw
dull laurel
@worthy cape about the path specification. is that the link between path_reps and the path_rep_bundle? and how do I get the link between a file info and a path?
worthy cape
@dull laurel The relationship between file_info and path_rep_info is unknown, but both has the unk[2] pair that may relate them somehow.
A path_rep_info denotes slices of the decompressed payload of the path_rep_bundle, which can be expanded into a set of paths by running the algorithm.
There is also another field in the path_rep_info of unknown meaning.
timber path
i.e. we don't know which file has which name, based on the path_rep stuff?
worthy cape
Correct. My goal after work today is to build a table of known file paths by comparing contents, one for each file entry.
Super slammed this week so it's going to be rough.
velvet fog
are there any other plans? I don't know what to do next.
timber path
Correct. My goal after work today is to build a table of known file paths by comparing contents, one for each file entry.
@worthy cape I am planning to do the same. Since the TXT has been removed, and my lib heavily relied on that, I have to look into this path_rep stuff now. I've ignored it so far because I accepted the use of the txt as the way-to-go; apparently it's not. So it's time to go back to the drawingboard.
worthy cape
@velvet fog Can you see if there's any link between the two unk u32:s in file_info and path_rep_info, maybe? Or see if the value that's not the offset/size means anything?
velvet fog
no idea :<
dull laurel
@worthy cape i compared them u32 wise for the two unknown in files and the three in paths and there were only two dozen matches in all those 17000.
worthy cape
ok, great.
velvet fog
I was thinking it maybe murmur2_64, because ggpk use tolower+ucs2-le+murmur2_32 for filename
vapid pulsar
I've observed fnv1 hashing in the client. not sure if it was for internal lookup though
dull laurel
so file and path are indexed by hash?
dull laurel
oh, ah, then those refer to each other and not file info to path rep
dull laurel
@velvet fog which part of index is path_spec?
timber path
What is path_rep actually used for? as in, what is its relation to the path_rep_bundle's decoded content?
dull laurel
path_rep points to a slice of path_rep_bundle's decoded content.
this slice can be used to produce a path using the path spec encoding section below (with the base and generation phase, though I haven't understood that yet)
worthy cape
The slice of the uncompressed bundle describes a series of strings with shared prefixes. The contents are control words that say how things are merged and string literals that are used by those control words.
timber path
Ah, so I can't just chain the entire decoded content? But have to adhere to the payload offset and sizes?
worthy cape
You can concatenate slices and still get the correct output as all slices reset the state at the beginning.
I don't know if the slices are consecutive.
dull laurel
So i decoded the first path rep and it contains multiple file names, which is confusing.
00-00-00-00-01-00-00-00-41-72-74-2F-32-44-41-72-74-2F-53-6B-69-6C-6C-49-63-6F-6E-73-2F-70-61-73-73-69-76-65-73-2F-41-73-73-61-73-73-69-6E-2F-34-4B-2F-00-00-00-00-00-01-00-00-00-44-65-61-64-6C-79-49-6E-66-75-73-69-6F-6E-2E-64-64-73-00-01-00-00-00-45-6C-75-73-69-76-65-2E-64-64-73-00-01-00-00-00-4E-6F-78-69-6F-75-73-53-74-72-69-6B-65-2E-64-64-73-00-01-00-00-00-53-6D-61-6C-6C-4E-6F-64-65-2E-64-64-73-00-01-00-00-00-54-6F-78-69-63-44-65-6C-69-76-65-72-79-2E-64-64-73-00-01-00-00-00-55-6E-73-74-61-62-6C-65-49-6E-66-75-73-69-6F-6E-2E-64-64-73-00-01-00-00-00-41-6D-62-75-73-68-2E-64-64-73-00-01-00-00-00-41-73-73-61-73-73-69-6E-61-74-65-2E-64-64-73-00
Art/2DArt/SkillIcons/passives/Assassin/4K/βΊadlyInfusion.dds
Elusive.dds
NoxiousStrike.dds
SmallNode.dds
ToxicDelivery.dds
UnstableInfusion.dds
Ambush.dds
two mistakes still there. that funny emoji (utf8, probably not utf8 :D) and the base path belongs to all filenames.
worthy cape
Each slice generates multiple full paths.
In the generation phase, the integer tells which string in the template set it should prepend when outputting.
dull laurel
but how to correlate those paths with the actual files inside the bundles?
worthy cape
That no-one knows.
I'm gonna conjure some dinner and then figure out some filenames but that means modifying some of my older codebases.
dull laurel
@worthy cape nice writeup, now I understand it.
worthy cape
Some slices have an empty template section and as such, always generates the given strings as-is.
Some slices have multiple sections, going template->generate->template->generate, clearing the reference set between.
dull laurel
00-00-00-00-01-00-00-00-41-72-74-2F-32-44-41-72-74-2F-53-6B-69-6C-6C-49-63-6F-6E-73-2F-70-61-73-73-69-76-65-73-2F-43-68-61-6D-70-69-6F-6E-2F-34-4B-2F-00-01-00-00-00-41-6E-45-00-01-00-00-00-49-6D-70-61-6C-65-50-61-73-73-69-76-65-00-00-00-00-00-01-00-00-00-43-6F-6E-71-75-65-72-6F-72-2E-64-64-73-00-01-00-00-00-55-6E-73-74-6F-70-61-62-6C-65-2E-64-64-73-00-01-00-00-00-57-6F-72-74-68-79-46-6F-65-2E-64-64-73-00-01-00-00-00-49-6E-70-69-72-61-74-69-6F-6E-61-6C-2E-64-64-73-00-03-00-00-00-2E-64-64-73-00-03-00-00-00-4E-6F-74-61-62-6C-65-2E-64-64-73-00-01-00-00-00-46-69-72-73-74-53-74-72-69-6B-65-4C-61-73-74-46-61-6C-6C-2E-64-64-73-00-01-00-00-00-46-6F-72-74-69-74-75-64-65-2E-64-64-73-00-02-00-00-00-46-6F-72-74-69-66-79-2E-64-64-73-00-02-00-00-00-54-61-75-6E-74-2E-64-64-73-00-02-00-00-00-41-74-74-44-61-6D-61-67-65-2E-64-64-73-00-02-00-00-00-41-75-72-61-2E-64-64-73-00
this slice is odd. it defines the template 01 twice, but uses a number 02 and 03 later.
happens to be the second path spec.
lean spindle
Was just looking at that bit as well, actually. Am quite confused atm
worthy cape
In the template phase, if a number has already been used, it pushes the contents of the number with the new string appended onto the set.
dull laurel
ah, so it automatically generates templates 2 and 3?
lean spindle
Wait, so if you have
01 00 00 00: Art/2DArt/Atlas/
01 00 00 00: Atlas
what is the resulting set?
Something like this? or am I misunderstanding entirely?
01 00 00 00: Art/2DArt/Atlas/
02 00 00 00: Art/2DArt/Atlas/Atlas
The numbers refer to strings in the template set, regardless of if you're in templating or generation.
If that is in the template phase, then yes, you have two templates at the end like that.
lean spindle
Ok, thanks for the help! (Also, yes, I meant template phase, I should have been more specific)
worthy cape
I'm still working off the original tech patch, but this is the interpretation of the second section: https://gist.github.com/zao/58e0d095dd5eb8e157273ad79b2560a6
(screwed up the reference indices, silly one-indexed things, reload)
lean spindle
I see, makes sense. Ty for the example, it makes things quite clear o7
worthy cape
Just slapped a ton of fprintfs into my existing decoder algorithm π
simple ravine
I am re-learning some C++ π
lean spindle
I'm learning c++ I never knew :P
dull laurel
i'm doing the same with C#
worthy cape
Uniform initialization gets around some parsing ambiguities where you can't tell if you're invoking a constructor or locally declaring a function.
simple ravine
you have an example? not sure I understand how that would look like
or I guess this quick tutorial will show me soon enough
worthy cape
simple ravine
scrolling thru some primer
worthy cape
It also has more special meaning when you use initializer-lists.
simple ravine
I was thinking that, it is likely more used in those situations where you have lists yeah
In c++ vector = array, right?
worthy cape
A std::vector is a dynamically sizeable dense array, like C# List.
A std::array is a fixed-size dense array.
simple ravine
A List but packed like array it sounds like
worthy cape
I'd be very surprised if a CLR didn't use dense storage for a List.
Possibly even mandated.
timber path
Why is the path_rep_payload_size two times in the path_rep? once as payload_size, and once as unk4?
simple ravine
well, an array is a little special
worthy cape
@timber path It's not always equal.
timber path
What did we learn from the cases where it's not equal?
worthy cape
Nothing yet.
I tallied the different values to see if there was some meaning: https://gist.github.com/zao/df89fdd4091f32c1af55a76bd96dab63
Second column is number of sections that had that value for unk4.
(this is over all path_reps, not just those that differ)
I'm currently being held up by that my old GGPK tools are incredibly slow.
timber path
path_rep 6300 gives me some buffer too small exception, which is odd... especially since the section contains 19762 bytes, and the buffer issue appears when reaching byte 2400
worthy cape
Is this 3.11.2.1 or 3.11.2.2? (regular or b)
worthy cape
I should re-export them, still working off the database exported from original.
timber path
I guess I shouldn't use while(binaryReader.PeekChar() != -1) are equivalent for your c++ while (r.n_) {
worthy cape
Mine is "while I have not reach the end of the block", so whatever analogue you have there.
timber path
Yeah, PeekChar returns -1 when it reached the end of the stream
But some peeks seem to return a value that isn't representable by a char (hence the exception)
I've replaced it with a less elegant while(pathReader.BaseStream.Position < pathReader.BaseStream.Length)
simple ravine
I normally try to avoid the Readers
timber path
Did you notice that some path_rep indices contain 0 results?
worthy cape
yup
timber path
index 19 is the first one with 0... why O.o
worthy cape
Not quite sure what to make of that.
Fun fact - putting all the exploded files in a single directory was a bad move.
timber path
Strange how file & path array lengths are so widely different
Like some paths correspond to multiple files?
Or are we maybe missing a layer of depth?
worthy cape
I'm missing the intended use of these structures. What motions does it go through when it needs to find a file with a given full or partial path.
timber path
Do we have a 010 Editor Template for the decoded paths bundle?
worthy cape
Not that I know of.
dense shore
sry wrong chann
timber path
I got this result for the first section of the paths bundle, its offset is 0, size is 334 ; When "decoding" it into proper paths, this was the result. It looks incorrect, based on the index 0 which is Art/.dds that makes no sense tbh
Maybe it is correct though π€ ... This section is taken from the old _.index.txt
lean spindle
Reading the file myself, from my current understanding of how this works, that looks right, actually
worthy cape
@simple ravine Hashes of all files in 3.11.1.7: https://zao.se/~zao/poe/3.11.1.7/sorted-digests-3.11.1.7.csv
worthy cape
No wonder this was slow... my "extract file from bundle" duplicated the whole uncompressed bundle in whole for each file entry. Also explains why I'm short a few hundred gigabytes of disk.
worthy cape
dump_file(output_path, bundle_mem.get(), BunMemSize(bundle_mem.get()));
vs.
dump_file(output_path, bundle_mem.get() + offset, size);
simple ravine
heh
so much weird code in this kraken.cpp
learning c++ by reading and trying to understand this, is like trying to learn how to swim by jumping into a pool with sharks
i.e. not healthy
#define ALIGN_16(x) (((x)+15)&~15)
#define COPY_64(d, s) {*(uint64*)(d) = *(uint64*)(s); }
#define COPY_64_BYTES(d, s) { \
_mm_storeu_si128((__m128i*)d + 0, _mm_loadu_si128((__m128i*)s + 0)); \
_mm_storeu_si128((__m128i*)d + 1, _mm_loadu_si128((__m128i*)s + 1)); \
_mm_storeu_si128((__m128i*)d + 2, _mm_loadu_si128((__m128i*)s + 2)); \
_mm_storeu_si128((__m128i*)d + 3, _mm_loadu_si128((__m128i*)s + 3)); \
}
like... wat
dull laurel
i'd say copies 64 bytes using mmx or some other vector operation? π
simple ravine
intrin.h yeah u're right
CPUID Flags: SSE2
Store 128-bits of integer data from a into memory. mem_addr does not need to be aligned on any particular boundary.
What does it mean when you have the asterisk like this (KrakenLzTable*)scratch
dull laurel
its a pointer. in this case he is casting his scratchpad memory (just some junk area) to a pointer to a krakenlztable
meaning, he stored a pointer to the table there earlier. I guess the algorithm tries to be very memory efficient, not doing many allocs / heap stuff
simple ravine
I'm guessing that *(uint64*)(s) then tries to cast whatever is in s to an uint64 by using pointer witchcraft
worthy cape
Reinterpret whatever kind of pointer/array s is into a pointer to a 64-bit integer. Then dereference that to get a 64-bit integer.
The trick relies on that the storage for s and d is aligned. If on x86, you may get a performance penalty. On cooler systems, you get a nice BUS ERROR and die horribly.
simple ravine
that doesn't sound cool at all
worthy cape
That was the best take-away we ever had on the network comms course at uni, most people started development on 32-bit x86 machines and had to port their code speaking the same wire protocol to 64-bit big-endian SPARC machines which barf on unaligned accesses.
simple ravine
For structs, the alignment is usually the maximum alignment of any member.
Ah, now I understand why game developers etc sometimes don't use arrays of structs, but rather have multiple primitive arrays for each member
worthy cape
The first element of a struct must begin on the first byte of the object, and as such the struct is aligned suitably for the first element.
If subsequent members would be misaligned, the compiler may insert padding after the preceding member to suit the member.
The final member of the struct may be padded to ensure that if the struct densely packed in an array, the next object has the correct alignment.
@simple ravine Not to mention ordering data members in order of decreasing size/alignment to minimize struct size π
Say that you've got a struct S { u64 a; u8 b; u64 c; u16 d; };
That struct will be 32 bytes large.
simple ravine
yes that is what I was attempting now in C++
struct Foo {
int a;
double b;
};
struct Bar {
double b;
int a;
};
int main()
{
cout << sizeof(Foo) << endl << sizeof(Bar) << endl;
}
8 bytes for the first member, 1 byte for the second member and 7 bytes of padding. 8 bytes for the third, 2 bytes for the fourth and 6 bytes of padding.
worthy cape
What's your reasoning?
How big would they be if it was tightly packed without regard for padding?
simple ravine
that double is larger than int and there should be padding between, or am I mistaken here
worthy cape
Padding may occur after a member to ensure that the next member or next object is aligned.
simple ravine
right, of course 
dull laurel
So I generated all filenames which comes out at 515383, but in the index there are 515438 files declared.
dull laurel
https://gist.github.com/totkeks/a52a72db1aa751c557afdd4b21adc006
all the data files are in the same path group (path spec?)
worthy cape
Don't know.
worthy cape
@dull laurel 8 sections have paths that end with lowercase(".dat")
PathSpec 21943: unk = {52B41254h,42BF9D27h}, unk1 = 7019143, unk3 = 29862, unk4 = 29862
PathSpec 21944: unk = {9F79B48Ah,06A751C0h}, unk1 = 7049005, unk3 = 29862, unk4 = 29862
PathSpec 21945: unk = {A5EEB780h,3A2B1C52h}, unk1 = 7078867, unk3 = 29862, unk4 = 29862
PathSpec 21946: unk = {C40365D5h,215CB17Ch}, unk1 = 7108729, unk3 = 29866, unk4 = 29866
PathSpec 21947: unk = {80BC12ADh,491548EDh}, unk1 = 7138595, unk3 = 29863, unk4 = 29863
PathSpec 21948: unk = {18CE0CB4h,D234EC55h}, unk1 = 7168458, unk3 = 29863, unk4 = 29863
PathSpec 21949: unk = {E9A30A7Eh,96A305BBh}, unk1 = 7198321, unk3 = 29860, unk4 = 29860
PathSpec 42119: unk = {E457FB13h,33D084A6h}, unk1 = 6989244, unk3 = 29899, unk4 = 238937
what's unk3 and unk4? those are equal for once
worthy cape
unk1 is offset, unk3 is size, unk4 is often equal to the size but sometimes not.
(I still have this old definition for the path_rep_info:
struct some_info {
uint32_t unk[2];
uint32_t unk1;
uint32_t unk3;
uint32_t unk4;
};
PathSpec: unknown1(E457FB13), unknown2(33D084A6), unknown3(0003A559) and 2526 paths
FileDeclaration(BundleDeclaration('Data.bundle.bin'), pos 0-35854 of 35854), unknown1 = 57AA215F, unknown2 = E19430CF
supposedly there is only 1 file in that bundle. and that file needs to point to a single path. so it needs to contain a reference to a path group and then inside of that to the correct path.
worthy cape
Behold: Art/Textures/Environment/Cave/treasure_pile@0017173Eh_100080h.raw also known as Art/Textures/Environment/Cave/treasure_pile/dog_colour.dds: https://i.imgur.com/5sarGzQ.png
I just realized that all the DDS textures I find by hash matching must not have changed between game versions.
worthy cape
Exported all file slices that began with DDS - https://i.imgur.com/gNxh6zl.png
Squinting at them, it seems like the colour of most of them is fine, but it could be that the vast majority are not BC7 too.
cosmic saffron
worthy cape
Most of the normal maps are BC7, as are the greenish ones but I don't know what PBR channels they are supposed to be.
mortal bone
Off Topic, I am finally working on this haha
Name: ShaderCacheD3D11.packed, Number of Entries: 61
Name: ShaderCacheVulkan.packed, Number of Entries: 125
Name: Bundles2, Number of Entries: 128
Name: Audio, Number of Entries: 2
Name: Bundles, Number of Entries: 3
Name: FMOD, Number of Entries: 1
The green textures are green in 3.11.1.7 as well, phew.
@mortal bone Ooh, GGPK shenanigans?
mortal bone
Yeah, I wanted to get to a point where I can mess with the bundle stuff
I am pretty much just following your data structures haha
simple ravine
I was just gonna ask, those normal maps looks a tad weird, but that might just be me
worthy cape
This is good, I can rest easier around textures now, instead focusing on the path mapping problem.
simple ravine
yeah I'm not gonna waste my time trying to convert this to C#
feels like a huge undertaking
vapid pulsar
Yeah POE just packs the normal coordinates on different channels. So the normal maps looks green haha
simple ravine
ok, who wanna nuget package that ooz stuff β€οΈ
i give up
too much double pointer dereference and hardware intrinsic wizardry
velvet fog
use ooz as shell execute
worthy cape
I'm going to hide in bed soon, but does anyone have any idea on how to use the known paths and possibly file type detection to connect file entries?
simple ravine
no sorry, zao... i have been trying to catch up with the black wizardry of ooz code. gave up.
velvet fog
nope :<
simple ravine
will probably dig, but if you haven't been able to, i doubt i will
how did u make the oozlib, zao? do u have that available somewhere or just threw it together quickly?
cursive sigil
ooz code made me cry
worthy cape
@simple ravine Plain boring DLL export of this - https://github.com/zao/ooz/blob/master/kraken.cpp#L4183-L4188, CMake project so just build and P/Invoke - noting that dst needs to be 64 bytes larger than dst_size.
worthy cape
My own code is in several tools that use libbun, my C tech-patch library that deals with bundles, which in turn depends on either libooz or a foreign DLL.
That code is not up anywhere yet as I'm still evolving the interface.
simple ravine
Ok
worthy cape
There's a pun in there, as it produces a libbun.dll π
simple ravine
π
simple ravine
I don't fancy hinging on 3rd party libraries that don't have a huge contribution/investment base, but don't see any other options here
oh well
so byte[] decompressedContent needs to be 64 bytes larger than int decompressedSize then I guess?
worthy cape
Yup, I've seen several of the blocks write up to half a dozen bytes out of bounds.
simple ravine
guess one will have to shave those off later then
worthy cape
(I overallocated and put sentinel bytes in the way that I verified after decompression)
simple ravine
zao, do you have a decoded _.index.bin handy?
just want to verify results once done
worthy cape
Which patch?
worthy cape
@simple ravine https://zao.se/~zao/poe/3.11.2.2/
simple ravine
merci bocoup
cursive sigil
beaucoup π
simple ravine
mercy bocko
worthy cape
As the names hint, one is the index, the other is the nested tail.
simple ravine
_.index.mem is the decoded _.index.bin I presume, in it's "raw" form
worthy cape
Decompressed blocks concatenated, as Dog intended.
simple ravine
Dog?
worthy cape
(amusing permutation of God)
simple ravine
π yeehaw it works
compact isle
hello friends, I should be able to do a post with API changes for Heist today or tomorrow
most significant is probably that we're switching to fmtlib from boost internally for formatting strings so some of that leaks out into the website world
simple ravine
hello friend, please leak some file specification for the bundles π
compact isle
that stuff is quite beyond me I'm afraid π
worthy cape
Yay for fmtlib, it's swell.
simple ravine
tell them i bribe them with swedish chocolate. i'll send some.
@compact isle, now I imagine you saying "Hello friends" like Scott Hanselman
compact isle
googles
simple ravine
compact isle
I like him already
simple ravine
he's a very nice person π
dull roost
nvm, its not
elfin oak
How do we use the json file? I think if I remember right, someone last league made a post on how to put it into PoB?
hazy fiber
trinkets missing?
velvet fog
@carmine merlin ΰΌΌ γ€ ββ ΰΌ½γ€ LOCALIDENTITY TAKE MY ENERGYΰΌΌ γ€ ββ ΰΌ½γ€
elfin oak
β€οΈ @dull roost
dull roost
if it doesn't bug you about overwriting gameversions.lua its not in the right folder
minor solar
ty vath!
worthy cape
@timber path I'm a bit non-interactive atm, in a video meeting at work π
timber path
@worthy cape I'm just tossing around ideas & finding from last night, hoping someone who isn't at work might have use for it or bounce back with their insights. During the day (CET) I have to work too, so I'm just lurking and not coding :p
rain pasture
anyone know what this [DNT] thing is in the passive stats?
"name": "Breath of Lightning",
"stats": [
"20% increased Lightning Damage",
"15% chance to Shock",
"[DNT] 20% increased Duration of Lightning Ailments",
"20% increased Effect of Lightning Ailments"
]
something that will change between now and launch then?
carmine merlin
3.12 tree update is live
rain pasture
that's a lot of damgae
timber path
Stupid question, but how do I become a recognized tool developer?
simple ravine
@hazy fog
Petition to add @timber path as tooldev.
inland kestrel
@mortal bone are there any instructions (or old commits) on how to update poeskilltree.com with the new passive tree info?
inland kestrel
Ah, okay. Yeah, it just came very recently. I didn't want to unnecessarily burden you but I've always like using that site for comparing versions π
mortal bone
inland kestrel
403 Forbidden π’
mortal bone
Yeah, uploading the stuff
dull laurel
any news on the filenames matching?
worthy cape
@dull laurel We've established that there's a common prefix to all paths in a single section, and all sections have an unique prefix w.r.t. each other.
In other words, they're directories.
mortal bone
@inland kestrel https://poeskilltree.com/?v=3.12.0-pre&c=3.11.0
dull laurel
ah, that explains why in the data folder there are over 200 entries and then there are small ones like this?
PathSpec: unknown1(810F821F), unknown2(3906914932), unknown3(216)
Art/2DArt/SkillIcons/passives/Assassin/4K/DeadlyInfusion.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/Elusive.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/NoxiousStrike.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/SmallNode.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/ToxicDelivery.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/UnstableInfusion.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/Ambush.dds
Art/2DArt/SkillIcons/passives/Assassin/4K/Assassinate.dds
I'd appreciate it if VS would crashing while writing code.
mortal bone
I have never had that happen haha
dull laurel
haven't used the big brother VS in a while. doing everything with VS Code so far. Works okayish for C#
simple ravine
I can't remember when VS crashed last time. It's super stable for me
hazy fog
Stupid question, but how do I become a recognized tool developer?
@timber path
If you think you deserve the tool dev rank and have something publicly available in use to point to: ping me with a link to it and what it does.
@hazy fog
No nominations
hazy fog
Yes.
blah blah tooldev # noone counts @timber path author of https://github.com/WhiteFang5/PoE-Asset-Updater
worthy cape
Yay for @GreenFang.
mortal bone
lol
timber path
@simple ravine thx for the petition. And @mortal bone thx to link my repo. Also thx for the role @hazy fog . Besides that tool I've published linked by Emmitt, I'm also the top contributor & collaborator of the Community Fork of PoE Overlay (https://github.com/PoE-Overlay-Community/PoE-Overlay-Community-Fork/graphs/contributors)
hazy fog
ooh
simple ravine
@hazy fog bruh, petition isn't nomination π
viscid talon
o/
I have a little bit of free time, and I am not sure how to get onboarded on the latest changes with the file format, open decompression libraries, and where folks need help. Any good resources to share?
simple ravine
this wiki/poe org is gonna render quite useful, even more so very soon π
viscid talon
@simple ravine do you mean there are other devs that are willing to migrate their content over there? That would be a first: https://xkcd.com/927/
rain pasture
what kind of loser would join an org without a picture~
mortal bone
To be fair, a common project for this work would have been super helpful right now haha
simple ravine
I could add PoeSharp to the organization if anyone feels like collaborating
it's fairly performant
timber path
Does Unable to MapViewOfFile for chunk header or Unable to MapViewOfFile in MapChunk MapViewOfFile failed in MapEntireFile ring any bellws? MapViewOfFile is a C++ function it appears?
mortal bone
@simple ravine what does you time to recursively grab all the directories/files look like without exporting?
I am at ~400ms right now
simple ravine
Not sure, but mine's not optimized for that, but more towards accessing specified folder or file.
mortal bone
Ah, I see
simple ravine
The use-case was to open the GGPK and access the Dat files without having to go through unnecessary things.
mortal bone
Ah, I recursively go through the files as well as build the tree at the same time. I "just" need to add a nice way to do dat parsing as well as this bundle stuff haha
simple ravine
Feel free to poke around in my code, for inspiration or if you feel it's something you want to collaborate on
I think I will have to add back the notion of IFile and IDirectory into the new branch.
I wanted to simplify the codebase Γ‘ la YAGNI, as I felt it was a little too engineered the first time around, but now I kind of think my first thoughts were fairly good afterall
viscid talon
10 people in the org.
@simple ravine I see only one person and one project in https://github.com/PoE-Tool-Dev/ , are you talking about another org?
simple ravine
viscid talon
got it.
simple ravine
It is up to each member to mark organisation memberships as public. It's private by default.
mortal bone
yep
viscid talon
TBH, I don't really care about individual member memberships, it's code that matters to me. I have very little to browse and learn right there: only that wiki page, though it is well dense π
thanks for answering my questions.
simple ravine
tbh, it was the wiki that was the immediate need.
we'll see what happens after that Β―_(γ)_/Β―
i think low-ambition-no-strings-attached stuff that potentially grow into something is the best way for OSS stuff
mortal bone
I have looked at your code a bit, but I ended up doing my stream read a bit differently haha
simple ravine
That's understandable
I still want to optimize the underlying things, yet try to keep a clean API user surface
viscid talon
i think low-ambition-no-strings-attached stuff that potentially grow into something is the best way for OSS stuff
@simple ravine 100% agreed.
mortal bone
public static unsafe T Read<T>(this Stream stream) where T : unmanaged
{
var buffer = new byte[sizeof(T)];
stream.Read(buffer);
fixed (byte* b = &buffer[0])
{
return *(T*)b;
}
}```that looks fairly similar to mine
mortal bone
It appeared that you implemented ReadInt32, ReadUInt32, etc?
simple ravine
yeah, I am not a fan of BinaryReader
mortal bone
Ah, I see. You have a very similar function further down haha
simple ravine
yeah, like this
public static async Task<Memory<T>> ReadAsync<T>(this Stream stream, int elements)
{
var size = Unsafe.SizeOf<T>();
var mem = await stream.ReadBytesAsync(size * elements);
return mem.ConvertMany<T>();
}
allows me to define a struct, and read directly into that
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe T To<T>(this byte[] buf)
{
fixed (byte* b = &buf[0])
{
return Unsafe.ReadUnaligned<T>(b);
}
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe T To<T>(this Span<byte> buf)
{
fixed (byte* b = &buf[0])
{
return Unsafe.ReadUnaligned<T>(b);
}
}
like so
mortal bone
Yeah, the read function I posted will do that same
var header = stream.Read<RecordHeader>();
simple ravine
smart. yeah.
I should try the same, see if there's any difference in reducing the abstraction
mortal bone
In my case though, T does need to be unmanaged haha
simple ravine
ah shit, I see now I am using ReadUnaligned. I should probably change that to ReadAligned.
stupid mistake.
viscid talon
@timber path where does that dll come from? PoEAssetUpdater/libooz.dll in https://github.com/WhiteFang5/PoE-Asset-Updater/commit/d90584c09817c334b35a304bca7e7b85cc36e85a ?
simple ravine
zao made a thing based on the ooz library
he said his tooling/pipeline is a bit in a flux atm, so not having something official for it yet
worthy cape
timber path
@viscid talon it's Zao's... https://discordapp.com/channels/174993814845521922/175290321695932416/754708382581129216
viscid talon
Awesome π
worthy cape
Got around loading my mapping CSV - https://i.imgur.com/UlR5qNo.png
simple ravine
side note, u make me want to get a 4K monitor
worthy cape
Does someone have a recommendation for a programmer's calculator that's better than Win10 calc.exe and a bit more accessible than bc -l?
worthy cape
Something purpose made, not writing things that happen to be valid code. π
simple ravine
what functionality are you missing? just curious, don't really have a recommendation
viscid talon
well computations with python are just simple π
you don't need fancy things, except if you want to import math
but @simple ravine made a good point: what are you looking for? : )
worthy cape
Seeing the number in multiple bases including binary, bitops, while still also being capable of floating point math and not horrible at keyboard input like calc.
simple ravine
Ah, so Calc Programmer mode on steroids
tho, it's pretty cool that microsoft has this
viscid talon
I didn't even know there was this on Windows...
simple ravine
viscid talon
TIL π
lusty musk
I'm working on a userscript for the official site
that adds a new search field
and you can type textual queries into it, and it will fill the search form with the appropriate setup
confirming with enter just does an immediate "quick search", shift-enter adds it to the current search form but does not immediately search
as some examples, the following queries work and do exactly what you'd expect them to
500 pdps, 2hsword, not corrupted, 5l
shaper, helm, ilvl 84, not unique
ring, 60 life, 30 chaosres, 60 eleres, < 5c
rare, shield, open suffix, not crafted, 80 life, 80 res, 70 mana, 28 block, craftable (craftable being a custom shorthand for not corrupted, not mirrored, identified, etc)
offline, song of the sirens
just a headsup, but @ poeapp made one of those, and said something about it was the feature he spent the most time on UI wise, but the least used.
I think for something like that to be useful, you need some intellisense/autocomplete list etc
it requires a lot of memory recollection when used
viscid talon
I am setting up my environment for windows dev for poe overlay, and the fact windows-build-tools installs python2.7 in 2020 makes me very sad.
lusty musk
@simple ravine I am aware of it
but that's not for the official site, and was quite a specific limited syntax
I try to be as liberal as possible with accepting syntax
and possibly adding hover tooltips to known elements that give you the 'keyword' to put it in a textual search
simple ravine
If I do [StructLayout(LayoutKind.Explicit, Size = 60)] does that mean my struct will be read Unaligned anyway? hmm
lusty musk
@simple ravine reading the docs you can control alignment with StructLayoutAttribute.Pack
simple ravine
Well, the default should be reading something aligned, but consdering explicitly setting size and offsets, not sure if that foreces the CPU to read that unaligned
public readonly struct IndexBinHead
{
[FieldOffset(00)] public readonly uint UncompressedSize;
[FieldOffset(04)] public readonly uint TotalPayloadSize;
[FieldOffset(08)] public readonly uint HeadPayloadSize;
[FieldOffset(12)] public readonly EncodingInDecimal FirstFileEncode;
[FieldOffset(16)] public readonly uint Unk0;
[FieldOffset(20)] public readonly ulong UncompressedSize2;
[FieldOffset(28)] public readonly ulong TotalPayloadSize2;
[FieldOffset(36)] public readonly uint EntryCount;
[FieldOffset(40)] public readonly uint Unk1;
[FieldOffset(44)] public readonly uint Unk2;
[FieldOffset(48)] public readonly uint Unk3;
[FieldOffset(52)] public readonly uint Unk4;
[FieldOffset(56)] public readonly uint Unk5;
}
why are you setting fieldoffsets explicitly?
as far as I can see that's just a tight layout
simple ravine
because when I combined two structs into one, the values were not read correctly
but that could be because I did ReadUnaligned in my extension method instead of Read (aligned).
gonna try again
without explicit sizes
worthy cape
But Pack'd?
worthy cape
UncompressedSize2 isn't on a 8-byte granularity so it'll be padded if you don't do anything.
simple ravine
I first had it in two structs as per the 010 definition had it
worthy cape
simple ravine
I should do Pack=8?
worthy cape
=1
You want it to consider fields sufficiently aligned if their offset is a multiple of 1, i.e. tightly packed.
simple ravine
hm that didnt work
Pack = 1 apparently
By default, the value is 0, indicating the default packing size for the current platform
ah, because there are 5 uints there's 4 bytes that would be padded there
got it
simple ravine
| Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------- |---------:|---------:|---------:|-------:|------:|------:|----------:|
| TestOne | 17.85 ns | 0.124 ns | 0.174 ns | 0.0210 | - | - | 88 B |
| TestTwo | 18.51 ns | 0.207 ns | 0.297 ns | 0.0210 | - | - | 88 B |
TestOne = mine
TestTwo = emmitt
@mortal bone ^^
| Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------- |---------:|---------:|---------:|-------:|------:|------:|----------:|
| TestOne | 17.85 ns | 0.120 ns | 0.168 ns | 0.0210 | - | - | 88 B |
| TestTwo | 17.39 ns | 0.248 ns | 0.364 ns | 0.0210 | - | - | 88 B |
attempt 2
vapid pulsar
Does
Unable to MapViewOfFile for chunk headerorUnable to MapViewOfFile in MapChunk MapViewOfFile failed in MapEntireFilering any bellws? MapViewOfFile is a C++ function it appears?
@timber path
This is used by the client to map sections of the ggpk into the processe's address space. Then they can read files as if it was memory and not have the overhead of the buffered reader.
pseudo ocean
Hey @compact isle ,
apologize for the ping.
i know you are usually hanging around in the tool dev chat and work on the API and stuff as far as i have noticed,
Today while i was making some Links for Heist preparation i noticed a weird thing with a mod on pathofexile.com/trade
The corrupted modifier for jewels 1% reduced mana reserved is not searchable via the stat filters,
instead it is listed as #% Increased Mana Reserved in the stat filters.
I have attached 2 images to illustrate what i mean.
I was just wondering if this was intended (Because personally i was quite surprised when i had to search them this way),
or if this was an oversight that you might be able to fix/forward for fixing.
https://i.imgur.com/nwnrbwz.png
https://i.imgur.com/PDWWXNG.png
Sorry again for the Ping, if this is not your department could you please let me know where to send it instead (if its indeed not intended)
tawny agate
@pseudo ocean A lot of affix work this way, it's an increased affix with a negative value
pseudo ocean
I see, its kind of annoying for someone like me personally (Autism, takes things Literal/exact meaning)
so i thought it was an oversight
mortal bone
@simple ravine ah nice
I am happy to see that we are allocating the same bytes haha
I initially had marshal.sizsof, and I couldn't figure out why it was returning 1 byte for a char. I haven't looked at Unsafe.SizeOf. I assume most of the speed difference comes from span
lusty musk
@polar island why? I want it for personal use π
@polar island speaking of syntax
I found your setup a bit limiting
so I ended up biting the bullet and made a syntax that is separated by commas
but in return I get to be much, much more flexible
well designing the syntax you run into ambiguity
e.g. for me
life > 60, life 60.., 60 life, >60 life, life 60
these all get parsed as the same thing
so basically no matter what the user writes, it will get parsed
and if you separate keywords only by spaces, some stuff becomes ambiguous
I also intend to add a lot of aliases
2hsword 2h sword twohandsword twohand sword etc etc
these all would parse as the same thing
or !corrupted not corrupted -corrupted corrupted no etc
compact isle
@pseudo ocean better to email those sort of things (roryv@grindinggear.com), this channel is for tool devs doing tool dev things
pseudo ocean
Thank you, like i said i just noticed you being in here before doing api and trade things, my bad
compact isle
yep but my contribution to this channel is helping tool dev things π
pseudo ocean
cant blame a guy for trying ;P
keen dragon
I don't know if this is the right channel for it but is it possible to filter for experimental bases or do you need to list them specifically
compact isle
you need to list them
keen dragon
I see, thanks
compact isle
@earnest radish not sure how they expect you to find those, I'll bring it up
also 
so apologies for slow replies
velvet fog
gems with quality has 'Superior' in typeLine
compact isle
^ an oversight, gems with regular quality won't show the affix in your inventory but alternate qualities will
worthy cape
I'm heading out for the night, utterly stuck.
waxen juniper
http://prntscr.com/uiol6o
Can anyone tell me what all this means, and how can I continue to live with it?
vapid pulsar
@waxen juniperI don't understand what your asking? do you know to know how the index bundle is laid out?
waxen juniper
i need to package index.bin back to client ...
Do I follow the methods of packing chunks, or does it not matter?
timber path
Maybe https://github.com/PoE-Tool-Dev/ggpk.discussion/wiki/Bundle-scheme#bundle-index-format helps to understand the index.bin
vapid pulsar
The process should be reversible but there is no need to repackage data? Modifying the game is generally frowned upon
fickle yew
@compact isle did the api change post you mentioned come out yet?
compact isle
nope
been swamped getting that alternate quality table ready
not too much to report though really
fickle yew
Sure. Pre-launch anything is always busy. Was just curious.
mortal bone
She ain't pretty, but she's mine:
dull laurel
what is that?
velvet fog
ggpk viewer
mortal bone
Currently just a hex view of the files in the ggpk. That right panel should change based on the type of file and display the contents.
dull laurel
just checked my world clock ... 01:00 in Auckland. π²
I guess the api has no openapi spec?
rancid lily
Is there a working ggpk viewer somewhere on github? I was thinking about doing one in Rust as a learning exercise, but I need to see how it's packed.
worthy cape
@rancid lily https://zao.github.io/poe-doc/ggpk.html
It's a great fun experience to design the data structures in a way that pleases the borrow checker, I've got a rather capable private poe-rs repo I built a fair number of parsers in until I stopped using Rust.
mortal bone
Everything I needed to get that viewer up in running was from zao's page haha the individual data formats will most likely come from pypoe
worthy cape
Even if one doesn't actually use it for anything productive, it's super educational to implement a real-world format and understand what makes it good and not.
mortal bone
Yeah, that is really the only reason I am doing it. It is a pretty fun challenge, and maybe I can help more in the future when these problems come up
broken cloud
Hi, any item filter gurus here?
Added AlternateQuality True parameter that allows you to filter for Alternate Quality Gems.
Added Replica True parameter that allows you to filter for Replica Unique Items.
Would "AlternateQuality False" and "Replica False" do anything?
dull laurel
Odd question for one of the C# gurus here
ulong hash = FNV1.FNV1.FNV1a64(path.ToLower() + "++");
var file = referencedFiles[hash];
file.Path = path;
Console.WriteLine(path); Console.WriteLine(referencedFiles[hash].Path);
The latter is empty. When I access the dictionary referencedFiles I don't get a reference to my struct?
timber path
@dull laurel depends on the type of referencedFiles and/or file ... if it's a struct, then you have to "put it back" into the dictionary (because struct is no ref. type)
dull laurel
@timber path a class would be a ref type?
simple ravine
structs should be mutable in most cases
try using classes (or records if you use .net 5)
dull laurel
using .netc0re
timber path
Yes, a class is a ref type
dull laurel
oh. i am behind then.
simple ravine
.net core 2.1 had vast perf improvements, then they even better in 3.0/3.1... now 5.0 is even more blazing
dull laurel
i think the lib for the index and bundles is working so far, now I just need a simple cli to get those files out
simple ravine
they're really squeezing every bit of perf out
dull laurel
haha, performance optimizations come later.
wind garden
ohhh man, it's been a while
dull laurel
like how long?
wind garden
over a year?
dull laurel
welcome back to Heist!
wind garden
last played blight league
hit lvl 100 and decided to take a break and play wow classic
dull laurel
that wasn't a good choice π
dull laurel
@timber path do you use visual studio or visual studio code (vscode)?
timber path
I use VS2019
dull laurel
any idea how to implement command line arguments easily?
wind garden
yes
mortal bone
CommandLineParser is a good one
dull laurel
odd that I didn't fine that with googling for command line parsers for c#
mortal bone
dull laurel
yeah got it now after you gave me the name, but meant before when looking for one
dull laurel
the latter seems to be easier with cascaded verbs
dull laurel
well what I want is something like
cli.exe index <index.bin>
cli.exe index list-files <index.bin>
cli.exe index list-bundles <index.bin>
cli.exe index get-file <index.bin> <fileA>
verbs
simple ravine
you just create several of those Option classes, essentially
dull laurel
ah and then I cascade option with option?
wind garden
[Verb] and [Option]
simple ravine
you can chain them iirc
wind garden
[Verb("execute", HelpText = "Execute a command synchronously")]
public class ExecuteOptions : BaseOptions
{
[Option('c', "Command", Required = true, HelpText = "Command to be executed")]
public string Command { get; set; }
[Option('n', "NoRedirect", Required = false, HelpText = "Causes stdout/stderr to not be redirected")]
public bool NoRedirect { get; set; }
[Option('w', "WorkingDirectory", Required = false, HelpText = "Working directory to execute the command in", Default = "")]
public string WorkingDirectory { get; set; }
}
like this
dull laurel
ah, I don't make a separate command for each subcommand (list files, list bundles) but instead treat them as options?
wind garden
one verb, multiple options
dull laurel
@wind garden but I got 2 levels of verbs. first index, then the command there.
simple ravine
see in their Options.cs they have 2 classes, one for head, one for tail
I feel that is the clearest way to represent what you want to do
dull laurel
wind garden
@dull laurel it does support enums as switch parameters though
so you could put your 2nd commands in an enum and call that
cli.exe index -s list-bundles -f index.bin
there's comments as recent as aug 13
so at best it's a future feature
but doesn't support subverbs yet
dull laurel
I will try with the enum and see if it works, or just skip subsubcommands
dull laurel
https://github.com/totkeks/poe-tools first working version.
now time to clean up a little
wintry surge
tried running it and got a Dll import error with libooz - what folder do you run the cli from?
Actually, getting an invalid ELF header error on libooz.dll
dull laurel
the base folder
wintry surge
issue is probably that I'm running it under WSL, I'll try using windows
worthy cape
I have not tested libooz in hippieland, just Windows MSVC.
dull laurel
mh, it's written in c++, right?
worthy cape
Yeah, it's plain ozz built into a shared library with one export.
dull laurel
I wonder what would work better for a public github.
or maybe "copy" that code to c#
keen dragon
would the new file format benefit from something like ggpk defragmenter (to reduce the file size after updates)?
dull laurel
no, it doesn't have empty space. or well, it shouldn't have. it could
worthy cape
@keen dragon Bundles themselves, no. The GGPK is still slightly vulnerable to fragmentation but as the file count is a tenth now, less so.
wintry surge
works on windows thanks
keen dragon
I see, probably not worth investing into atm then thanks
dull laurel
ah lol yeah @wintry surge. won't work in wsl π
worthy cape
@keen dragon Every byte of the index has a purpose. Individual bundles have some fluff in the headers but nothing you can change. The compressed payloads are assumedly all meaningful data and there's already content-based deduplication in place for files that are identical.
keen dragon
I understand, I'm just wondering how aggressively they cleanup old stuff on patching or if it's even possible to know if something was deprecated
dull laurel
I'd say with the new bundles, those files are generated during their build pipeline and then replaced accordingly. So you should always get optimized files
simple ravine
hmm, torrent is potentially released today right?
tiny cargo
https://twitter.com/pathofexile/status/1304142342228570112
According to their schedule, "maybe"
Twitter
simple ravine
hence > 'potentially' π
worthy cape
There will be and we'll have a news post alongside it which tells you what to do with it.
simple ravine
What is this referencing to, @worthy cape?
worthy cape
"Will there be a torrent download?" https://www.reddit.com/r/pathofexile/comments/iutfvy/-/g5n29je/?context=3
simple ravine
I am weighing between using ref structs for the index bundle, to store the name as a ReadOnlySpan<char> or if I should just create a simple class and allocate the strings on the heap
worthy cape
Ooh, experimental macOS client up.
simple ravine
oh wait for the barrage of "y u no have mac option" on all 3rd party tools made for windows π
worthy cape
Currently fixing libbun to build on Debian. Most of the code paths for Linux are dry-coded π
simple ravine
ambitious π
rapid pagoda
@worthy cape mac client doesn't even run out of the box, quarantining issues π€¦ββοΈ
worthy cape
My work MBP is Mid-2011 and runs the highest supported release High Sierra 10.13. I don't have high hopes for it.
rapid pagoda
xattr -dr com.apple.quarantine '/Applications/Path of Exile' fixes it for now
worthy cape
Download doesn't even work now, but that could be my rotten Firefox install.
rapid pagoda
I got a Chrome warning about the download being from a non-HTTPS URL
worthy cape
I'm two versions behind on the OS required, this needs 10.15
rapid pagoda
I'm mostly just curious how hilariously my laptop will fail to run it. 2018 Macbook Air
(= teeny tiny puny integrated graphics)
worthy cape
This one has Intel HD4000, the core reason why they stopped releasing OSes for it was that it couldn't do Metal.
rapid pagoda
I'm curious whether GGG have access to a DTK, and how (if at all?) the game runs on there
worthy cape
I'm mostly curious as to how it's claimed that the PC GGPK will be useless.
rapid pagoda
The Mac GGPK appears to have a different PDIR structure.
Or possibly it's just in a weird state while it's installing. We'll see. π
worthy cape
I don't seem to have any surviving VMs with current-ish macOS, bleh.
rapid pagoda
The Mac patcher looks a little weird π
dull laurel
why? looks macish
wind garden
smooth
dull laurel
so, what kind of commands would make sense for a command line tool with the new bundles? currently can extract a single file by name. and you can get the list of all filenames.
worthy cape
bun_extract_file now builds and runs on Debian Linux.
Including liblibooz.so because I'm competent at CMake.
rapid pagoda
blibblibliblib.
worthy cape
I'm slightly upset that my pun of libbun.dll doesn't work as well when the ending is .so π
rapid pagoda
libbun.dylib is even sillier sounding, though!
dull laurel
guess I need to include libdat now, so I can do something with those files
rapid pagoda
β¦ okay, this looks really silly on a Retina display
worthy cape
A game for ants!
dull laurel
change resolution? disable the retina scaling and pick the actual resolution of the display?
rapid pagoda
⦠it also beeps every time I press Esc or Enter. So⦠uh⦠that's a thing.
rapid pagoda
β¦huh, okay. So I think the MacOS GGPK uses UTF-32 for filenames??
wtf why god why
dull laurel
maybe they must on mac?
rapid pagoda
Definitely not. I was interacting with the old style GGPK just fine before
And it's not like they even have non-ASCII filenames.
simple ravine
mutters something about game-devs...
vapid pulsar
Mac for sure supports UTF-8, 16 and 32
https://developer.apple.com/documentation/foundation/nsstring
rapid pagoda
And they didn't bump the version in the header either, so you can't distinguish between v3 Windows GGPKs and (incompatible) v3 MacOS GGPKs.
dull laurel
maybe there is a hint somewhere for the encoding`?
rapid pagoda
Once I've corrected for that, though, the general structure looks about the same.
worthy cape
It could very well be for ease of use with their exploded development trees, native filesystem functionality might be more ergonomic with UTF-32.
rapid pagoda
Maybe. Just weird that they wouldn't do it consistently across platforms.
worthy cape
I'm doing my best to not have anything to do with macs at work, blissfully unaware π
rapid pagoda
It's a VFS either way, no reason it needs to match local platform conventions
dull laurel
haha so many people with macs around me. all those new hipster developers
unreal socket
And they didn't bump the version in the header either, so you can't distinguish between v3 Windows GGPKs and (incompatible) v3 MacOS GGPKs.
@rapid pagoda I've told the programmers about this, they'll see what they can do after Heist released is sorted.
There is a macos channel in the Help section now btw
I'll be in there mostly
worthy cape
Time to hit the sack, hoping for a smooth launch tomorrow. Might even get to play the game π
simple ravine
good night, zao!
dull laurel
@mortal bone whats the plan with the github org btw? do you want to group all the poe related projects in there?
simple ravine
good question
I was suggesting a common place to put information regarding the ggpk bundle stuff. Not sure we'll use it for more things, but we'll see I guess.
daring moss
What's GGG's stance on using your poesessid to access information about your own account?
mortal bone
@dull laurel um, I originally created the org a long time ago to hopefully group some of the dev knowledge in a more central place. It would be nice to see a lot of our common problems/documentation there. For example, parsing the json data and all of the old data, or ggpk parsing and various formats. Similar to what zao has done
daring moss
Oh yeah I don't intend to lose it. Just wanted to know if they mind if someone requests their own stash every now and then
worthy cape
I'm quite happy to lift my other docs like GGPK there, and whatever other formats I've got under documentation.
@daring moss You've had tools that use it since the dark ages of forum trade like Procurement, I think it's pretty much "don't put it in things you don't trust" territory.
mortal bone
We could also put them in the repo and use GitHub sites
A bit more organized than the wiki haha
dull laurel
github wiki is fine too, isn't it?
worthy cape
I like the collaborative editing thing, I used mdbook for mine mostly out of curiosity.
mortal bone
Ah, ok
daring moss
are any of the "private" calls like fetching your own stash documented?
rapid pagoda
https://app.swaggerhub.com/apis-docs/Chuanhsing/poe/1.0.0 covers some of it, iirc
Most of it is pretty straightforward.
vapid pulsar
What is the current state of the oauth stuff?
broken cloud
Oh yeah I don't intend to lose it. Just wanted to know if they mind if someone requests their own stash every now and then
@daring moss as long as you don't do it too often (current rate limit is 45 requests every 60 seconds)
daring moss
oh that's way more than I ever thought I could do haha
was thinking one a minute or so max
I can imagine it being straightforward but if it is documented I don't have to pull out my browser's dev tools and figure out the endpoints giving me more time to focus on what I actually wanted to make
vapid pulsar
Darn, Im at work. cant poke at the torrent yet π
inland kestrel
I blinked for a couple of days. Does pypoe know how to open this now? or is there an alternative file extractor?
I'm looking to pull out the basics (as you'd expect): stat_translations, mods, base_types, etc etc
simple ravine
@inland kestrel https://github.com/zao/ooz
inland kestrel
β€οΈ
simple ravine
there are some other initiatives, check #tooldev-breakout and this
mine's not done yet
fickle tusk
Nah, you're just loosing your changes once pob updates
simple ravine
new vs old index bundle.
25000 more files lol
Data*.bundle.bin files are cleaned out
worthy cape
@velvet fog did you see the part in the torrent announcement where they are adding some league content a week or so into the league? Not sure if it messes with how you do things.
IMPORTANT NOTE FOR CHALLENGES: We're planning to introduce some Heist content after the initial expansion release, most likely by the end of the first week. This is noteworthy because some of these bosses are necessary for challenge completion.
velvet fog
no idea what they will change, but I will keep in mind, thanks
timber path
@simple ravine your index compare is torrent vs current? Or current+torrent replacements vs current?
simple ravine
new on left
dull laurel
the performance of my C# index extractor is abysmal. memory usage jumps between 500-1500MB and it takes nearly 6 minutes to extract 2526 data files from the index.
Is that a C# problem or is my code just bad, because of lack of C# knowledge?
simple ravine
I was looking at your code, and there are some low hanging fruit in terms of performance.
worthy cape
What's your approach? I sort by bundle, full decompress, and then slice out individual files per bundle.
simple ravine
I am writing my implementation, that you can take some inspiration from
worthy cape
2000s for 140k files.
simple ravine
I had to go to bed at like 5am but will do some stuff... In a meeting with Neotys atm lol
@dull laurel which license of VS do u have?
if you have Enterprise (maybe pro?) u have the Diagnostic thing which is pretty great
dull laurel
@simple ravine currently using vscode, but guess I have to switch to real VS due to c# support just being much better there
dull laurel
@simple ravine remember any of the low hanging fruits? otherwise I will try to parallelize the extraction process now. Just need to make the code thread safe
simple ravine
need to run out to an appointment, but will be back i na little while
i haven't gotten to the inner bundle song-and-dance or making dictionaries yet, but here you can see my approach so far.
worthy cape
@dull laurel My initial exporter was super dumb - for each file decompressed the whole bundle, sliced out the file and threw the rest away. Repeat π
Got way better when I sorted by bundle and extract all the relevant files per bundle.
It could get even better if I selectively decompress the blocks that are needed.
dull laurel
Yeah, I just thought where to put the thread locks or how to split in general. Either split the tasks by file, so one thread per extracted file or by bundle, so one thread deals with all files from a bundle.
The latter might be easier to implement
worthy cape
Depends on if you're doing batch operations known up-front or are more working on a streaming basis too.
It's nice that it's almost free to parse the index now, so the biggest cost is in scheduling decompression.
dull laurel
almost free compared to before with ggpk?
worthy cape
Slurping in the lists into memory to the point where you can look up a full path is almost instant, all you need to do is build a lookup table for the file entry list (or linearly search).
Looking up one file in a GGPK means hitting the disk for a chunk for every directory component comparing component hashes or names on every level.
Building a full directory tree in a GGPK means hitting the disk millions of times, which sucks even on a SSD.
worthy cape
However nasty the current scheme is to figure out, it's cleverly efficient. It's the pack system for the modern world π
dull laurel
the modern world uses javascript and is highly inefficient. source: I'm a web developer π
odd sail
almost reminiscent of a bloom filter
Data*.bundle.bin files are cleaned out
@simple ravine what does this mean?
simple ravine
In the torrent, they've removed the Data files.
odd sail
where did all the Data files go?
simple ravine
prevent pre-emptive data mining
dull laurel
hehe the usual π
worthy cape
@simple ravine So is it just the bundles missing/empty, or also the file info entries and/or path_reps?
simple ravine
I haven't checked if the bundle info is still there, I just noticed that the Data*.bundle.bin files were gone from the extracted ggpk
velvet fog
other files just fine
worthy cape
bundle info is still there for Data bundles.
file info is there as well, so just the actual bundle files then I guess.
worthy cape
Good to know how the redaction is done still.
Ooh, has anyone looked at the path_rep paths, see what new Dat files there may be?
simple ravine
I took a look at Bundle\Preload_x64.txt
it lists some new .dat64 files
Data/HeistBalancePerLevel.dat64
Data/HeistConstants.dat64
Data/HeistDoors.dat64
Data/HeistGeneration.dat64
Data/HeistIntroAreas.dat64
Data/HeistLockType.dat64
Data/HeistNPCDialogue.dat64
Data/HeistNPCs.dat64
Data/HeistRevealingNPCs.dat64
Data/HeistValueScaling.dat64
Doors!
worthy cape
There's more.
simple ravine
much likely yeah
just got back into trying to figure out the last part of the index with the file names
dull laurel
@simple ravine do you understand the difference between Span<T> and Memory<T>? I can't figure out which of those would be the right way for shared memory between threads (the decompressed bundle data in that case)
simple ravine
Span is stack allocated, so dont share that between threads.
worthy cape
@simple ravine https://gist.github.com/zao/2f999f62d40a2489bc4cdbf44f12ac52
worthy cape
Multiplied by four, but yeah.
dull laurel
why are they doing this? whats the difference between those 4 types?
worthy cape
@dull laurel 64 has wider pointer types, L supposedly has UTF-32 text.
simple ravine
32bit, 64bit, mac?
dull laurel
ah, so its easier to read and cast the memory?
because they don't have to convert the pointers?
worthy cape
Considering that the Mac GGPK supposedly has UTF-32 names for the heck of it, L is likely Mac-related.
Cleaned the gist list up.
Looks like the DAT smashers will have ample of fun tonight.
simple ravine
lots of RE to do still I see
fortunately the DAT structure itself is well-known, just need to understand the correlations
worthy cape
The good old kind of field searching, thankfully.
I'm gonna be hanging around at launch in case something surprising happens.
fickle tusk
I'm gonna be hanging around at launch in case something surprising happens.
@worthy cape like a bug-free league launch? π
dull laurel
you don't play normally? π
fickle tusk
for any specific reason or just burnt out?
simple ravine
got to lvl 100 in Legion, and since then, I haven't found it mechanically challenging enough to play
yeah, kinda
worthy cape
@simple ravine 40/40 last two leagues, level 100 this league.
This one will be more casual, I tell myself.
dull laurel
wow crazy
worthy cape
I made the most progress on my PoE viewer tool when I ragequit the game for a week over how all my builds sucked mid-league.
dull laurel
i think my highest was some 20 challenges
worthy cape
@obtuse citrus You want to overallocate the output buffer for decompression by 64 bytes while keeping dst_size the same, Ooz_Decompress writes out of bounds.
dull laurel
should write that somewhere in the docs
worthy cape
@dull laurel Mentioned in wiki now, already had it in my higher level bun library.
obtuse citrus
huh allright
worthy cape
Did some testing of it, saw it smash up to 5 bytes at times. Ooz define is 64 bytes, so 64 it is π
dull laurel
how did you notice it? c++ error? because in c# i don't get any
worthy cape
@dull laurel I overallocated the buffer, filled it with sentinels like 0xCD and then looped to verify it after decompression of a block.
dull laurel
ah π (OCD. haha)
worthy cape
Depending on how things are allocated, your allocation may be oversized from the runtime, and thus write into unused contents, or there's smash protection sentinels for hardening that the debug runtime verifies for you, or you may be on a page boundary and have an access violation.
Or the "best" case, you may have adjacent allocations that you (partially) overwrite and break the world.
Undefined behaviour - not even once π
simple ravine
I send off uncompressedSize + 64 but then I only take uncompressedSize when storing.
so far it has been successful, but i haven't tried a lot of files
civic crane
whats the path of Client.txt in Steam?
worthy cape
<library>/steamapps/common/Path of Exile/logs
cf. "C:\Games\SteamLibrary\steamapps\common\Path of Exile\logs\Client.txt"
I've got some code to find and parse the Steam files so you can go from the registry location of steam all the way to the PoE install dir.
Sweeps the Steam directory and all configured libraries for the PoE installation.
dull laurel
can't you access the registry for that?
worthy cape
@dull laurel You can determine the Steam directory from the registry. The game may be in a separate Library.
wintry surge
aha, got @dull laurel cli running on wsl. Got libooz built thanks to the Debian support added yesterday and got the .so linked apprioriately
dull laurel
using the .so from @worthy cape?
wintry surge
here's what I did to build it / get it running:
- installed libsodium, libunistring
- cloned zao's ooz repo and ran
cmake --build ., copied the resulting liblibooz.so to /usr/lib - changed the dllimport tag to libooz in projects/Bundles2/src/LibOoz.cs to
[DllImport("libooz", CallingConvention = CallingConvention.Cdecl)] dotnet run --project projects/CLI index --list-bundles <index file location>
worthy cape
You probably want -DCMAKE_BUILD_TYPE=Release if you haven't already, I forget what the default is. And yes, I'm naughty for not declaring the dependencies for the bun library.
You could get rid of those deps if you just keep the libooz bits.
wintry surge
I'll rebuild with that for extra speed lol
worthy cape
I'm tempted to rename the library so it doesn't have the pointless extra lib prefix, but don't want to do it this close to launch.
dull laurel
also why ooz?
simple ravine
license.
worthy cape
My choice over third party libraries is that it's more portable and I've got Linux boxes in my workflow at times.
Could technically also work on Macs, but I'm not booting that to check.
worthy cape
Probably.
dull laurel
then i don't have to integrate the source and stuff in my repo
worthy cape
Still probably should split off all my libpoe/libbun stuff from the ooz repo, had it there mostly to avoid having to deal with deps π
I'm going to add an additional export later that allocates its own storage and incurs a second copy, just to make life easier for people who don't want to deal with SAFE_SPACE.
dull laurel
that costs performance
worthy cape
Thus the "additional" π
dull laurel
would it be hard to migrate the c code to c# for libooz, so we don't need different libs per OS?
worthy cape
Lots of bit-twiddling to understand. You could probably mechanically port it assuming you grok the subtleties of all the bitops.
dull laurel
checked the original code. its 1 commit at most per implementation. 0 information π¦
mortal bone
It wasn't too hard adding 64 to the buffer haha
dull laurel
that's the easy part π
worthy cape
It's more having to know that it's something you have to do, and it's nothing the function can verify.
dull laurel
is there a license issue with re-implementing the oodle (de)compressions?
worthy cape
A bit twiddly to have to overallocate and cut the buffer size down for the caller, and if they supply their own storage, that detail leaks out too.
@dull laurel To be completely clean, you should be working from the documentation of a white-room reverse-engineering or analysis.
Then you just have the problem of patents <_<
dull laurel
what does that mean?
worthy cape
White-room? One party reverse-engineers something and documents it in enough detail that someone else can reimplement it from documentation. As none of the actual thing is communicated, copyright concerns are alleviated.
mortal bone
basically, both parties are clean because neither one of them really did anything that is bad
dull laurel
would be cool to have that dotnet stuff running on all dotnet supported OSes, hence the question about libooz
worthy cape
In this case, I don't think the author of ooz would care much at all, based on what I've heard about them.
simple ravine
Spent some time looking at kraken.cpp and all the shenanigans they're doing, and I felt it was just out of scope. Would take far too much time. Pareto rule.
kraken.cpp is > 4000 lines of code alone
dull laurel
and zero comments?
vapid pulsar
Maybe we can convince ggg to pull out oodle from being embedded into the exeand just include the dll.
dull laurel
or install warframe @vapid pulsar
simple ravine
that would still require an installation of Path of Exile, which in most cases is fine
vapid pulsar
Right but telling other people who want to use community tools to download warframe isn't the greatest.
dull laurel
haha true
worthy cape
Also needing to be on a supported OS.
dull laurel
that's why dotnet to the rescue. meh.
worthy cape
Rust, Rust, Rust π
vapid pulsar
Rust is love, Rust is life
worthy cape
@vapid pulsar Buddy could identify the debugger by "time wasted debugging:" alone π
vapid pulsar
Yeah haha
simple ravine
ok, a case
dull laurel
1 beer for 10k lines of code
worthy cape
Find one of those autotranslators and see how hard it botches it? π
simple ravine
to be fair, i think it'll be less than half in c# if you're streetsmart, and use Spans etc
vapid pulsar
make it a keg and maybe it will be worth it
worthy cape
The solution to everything is Fortran btw.
mortal bone
googles c to c#
dull laurel
span magic?
mortal bone
It might be a fun project tbh
dull laurel
i need to learn those spans first. currently upgrading my bundle / index extractor to work with multiple threads
simple ravine
Spans are essentially your gateway access to unmanaged memory without the need of pointers.
dull laurel
i'm starting to use them now instead of byte[]
mortal bone
it is like managed unmanaged memory lol
pearl sinew
does anyone have the preinstall for hiest
simple ravine
yes
dull laurel
my knowledge of C# is just very limited
worthy cape
Holy moly, I was going to mention how I ported the Serpent cipher from reference C and academic paper to Rust.
simple ravine
you kind of need a little bit of knowledge about memory and how it behaves, not zao-level knowledge, but a little bit
worthy cape
The RustCrypto people have looked at my implementation and made a proper one π
simple ravine
you mean they stole it and called it their own? π
dull laurel
i know how memory behaves. it's a condensator that stores electric current. and then there is signaling to read a line at a time π
vapid pulsar
Unrelated, but who gives tool dev discord roles?
worthy cape
There's a pin somewhere.
simple ravine
vapid pulsar
Sweet, thanks
mortal bone
technically I can as well lol
worthy cape
Don't want to take away from Ferret's enthusiastic counting.
simple ravine
he gave up the counting hehe
inland kestrel
Doesn't wrapping ooz in a p/invoke layer work well enough for now?
golden bane
Alright, I'll put down a 100β¬ (118 USD) bounty for full white-room documentation of Kraken, Mermaid and Leviathan decompression
Probably still not worth it, but maybe someone else wants to sweeten the deal as well π
dull laurel
nice! 100β¬ plus a beer. it's piling up
simple ravine
I'll throw in a box of chocolate and a teddybear
List<byte> rawString = new List<byte>(); // refactor for the love of god
π
wonder what the maximum string length is
probably not more than 256
perhaps defining a static buffer space of 256 and use that
even a StringBuilder would be better.
golden bane
@fickle yew Obviously low priority at league start, but you might want to update your FAQ now that you've switched to PoB Community
dull laurel
on demand decompression reduced my memory footprint by 1gig. that's odd.
fickle yew
@golden bane Noted. I'll get it updated. There's no way to support the fork right?
I'll match @golden bane's bounty offer.
golden bane
@fickle yew Nice! We might offer some way to indirectly donate to charity in the future. Also @carmine merlin is trying to become Twitch partner, so one could gift him their Prime subscription or something. But generally, we aim to keep money out of the project, since there is no way to distribute it in a fair way with this many contributors.
dull laurel
there is github sponsoring on issues
is he streaming the game or streaming PoB coding sessions?
golden bane
Mostly coding sessions and some gaming afaik
dull laurel
I always wondered if people are interested in learning from coding sessions instead of watching their favorite streamer play game.
fickle yew
I've considered streaming a coding session or two. But haven't tried it yet.
golden bane
It's mostly just a place for our users to hang out and catch up with development. Sometimes, other contributors join the voice chat and answer questions, etc.
As with most streaming, it's mostly not about the game, but the person(s) behind it. So I wouldn't worry too much if the task at hand is interesting or not.
fickle yew
Ah no chance for me then I guess π
dull laurel
not a people person @fickle yew? π
fickle yew
Well he says I need to be interesting
I do feel like I spend 80% of my work time on zoom though so maybe I'm already partly a streamer.
simple ravine
I'm too slow and distracted of a coder nowadays, would be utterly boring to watch me lol
fickle yew
I'm actually writing more code than I would have thought in my current position, which is positive π
fickle yew
Ouch
dull laurel
I got a little more, but got pushed into a product owner role. less coding. more jira-ing π
simple ravine
Ouch.
fickle yew
I've made... 5 pull requests today.
simple ravine
daym
fickle yew
Small ones though.
simple ravine
PRs should be small π
fickle yew
What do you do all day then @simple ravine ? Hang out in an ivory tower?
worthy cape
"draws bubbles and arrows"
fickle yew
I've drawn the box where the Kraken algorithm should be implemented. Go go minions.
dull laurel
ivory tower = software architect?
fickle yew
Yeah
golden bane
@fickle yew I didn't want to put you in a bad mood, sorry! Not everyone wants to be an entertainer and that's fine. I know I wouldn't π
simple ravine
my title is long and stupid
fickle yew
I was mostly kidding @golden bane
fickle tusk
just watching the stats page on poe.ninja is pure entertainment - no personal interaction needed
dull laurel
@simple ravine sounds like you work in the same company as I do
golden bane
Alright I couldn't tell and thought I just spoilt your day π
fickle yew
Yeah it's hard to pick up on text. Sorry about making you feel sorry about making me feel bad.
simple ravine
@dull laurel I work in one of the larger consultant companies. My role is Chief Cloud Architect for the Swedish unit, which is part of our Rightshore & Large Deals unit
dull laurel
oh okay, but aren't those titles rather short? "Director", "Associate", "Senior ..."?
simple ravine
for most people they are yeah, like 'Consultant', 'Senior Consultant', 'Managing Consultant' etc
anyways, back to the path mangling of that damn index
dull laurel
how can i thread-lock access to a certain dict/list item? I want my threads to wait if they access the same bundle the first time, so one thread can read the main parts.
simple ravine
you don't
you use the System.Collections.Concurrent variations
such as ConcurrentDictionary
fickle yew
Or look at Lazy if you only want to protect initialization