rich quarry
So I modded P3P Steam without issues, but I noticed that when I use even one mod loading times are doubled and AOA/Co-op attacks stutter A LOT
This happens if I use even just Controller UI Overhaul or Cleaner Bustups, and no, my pc is not the issue, I've ran heavier games, as well as P4 and P5, fully modded and I never encountered such issues
No, both the mods and the game are on C:
rich quarry
Alright, gonna do that now
rain hedge
The mechanism used for redirection in P3P, P4G and P5R is identical.
There's probably an external factor in play if something unusual is going on.
I can't exactly say I'm sure what it could be off the top of my head. Is anything being actively logged during gameplay?
rich quarry
I can't exactly say I'm sure what it could be off the top of my head. Is anythin...
Nothing is being logged during gameplay
rain hedge
Quite honestly, for now, I have no idea; you can set Log Level to Debug in CriFsHook [the mod that does the actual redirection] and see if you get anything more.
From my perspective, my code isn't doing anything once the binding is done; it's all on the game; and if that's being slow for some reason, even with a small mod set; it's just really, really strange.
Especially if it runs fine with P4G-64 and P5R; it's the exact same code
rich quarry
Quite honestly, for now, I have no idea; you can set `Log Level` to `Debug` in C...
I don't know either, I'm not at home right now but I'm gonna do some more testing tonight, see if I can find what's causing this issue
It's especially weird since all the mods I tried don't affect battle at all, yet that's where both the stuttering and loading issues appear, nothing wrong in the overworld
rain hedge
If it's only battle; then that is really, really strange.
inb4 Denuvo
Note: Enable Print File Access in CriFsHook; if it's not printing anything during times it stutters; it's not mod related.
digital ether
Note: Enable `Print File Access` in CriFsHook; if it's not printing anything dur...
was noticing the same thing too, but seeing stuttering during overworld, wasn't printing in the first room, which didn't have stutter, but outside it was
rain hedge
Oh dear lord
rich quarry
Yep, my log printed exactly the same thing, I'll attach it down here
None of this is printed if I remove Persona Essentials (which I left running without other mods other than CriFs and Controller Button Prompts)
I noticed that running Persona Essentials and CriFs alone doesn't print this, so I'm guessing it's got something to do with when there are other mods, as this happens with
Controller UI Overhaul
P3HD WIP
Cleaned Up Bustups
(it might happen with other mods but I only tested those three - one by one)
digital ether
still noticing it with only crifs and persona essentials
\tex2_design\field2d\icon\ICON01\000.TMX
\tex2_design\field2d\icon\ICON01\001.TMX
repeated
with stuttering in fields, although it stutters harder with mods
rain hedge
Can someone give me a save game?
I haven't even started game
I'm not sure if there's anything I can do about it, but I can try
I might need to play through intro anyway, oddly it's not picking up the saves
in userdata\<my_id>\1809700\remote
it looks like a long hash lol
back to holding Y I guess haha
Can you grab your p3p.ini?
Maybe it's a flag
rich quarry
was noticing the same thing too, but seeing stuttering during overworld, wasn't ...
nothing wrong in the overworld
He said,
yeah now that I checked better, it's also in the overworld, same thing as the guy I quoted
rain hedge
yeah maybe it checks Steam account or something
it ignores any save that isn't mine
typical Japanese devs I'm guessing
f
digital ether
not sure why if converting saves works
the tutorial for that at least says the character name/gender needs to match between the two
rain hedge
well, that would maybe help explain
rain hedge
It might be tied to the other file Pixel uploaded
I put it in wrong folder since I didn't know where at first
ah there we go
well, kinda go, only slot 1 is enabled in save menu
seems something somewhere else controls that
but I can just rename file, it's okay
I can't currently reproduce :/
Saves I got from @obsidian knoll are in tutorial dungeon
issue seems triggered by repeated menu file load meanwhile
oh I guess I can try All-Out-Attack in the meantime
yeah, a cleaner repro would be nice π
original video shows it caused by repeated hub/menu file load
I did actually see that triggered in school area
before first save point
but it takes 2-3 minutes to get there
which not good for testing
digital ether
i have some files, slot 2 should be able to do everything
rain hedge
oh yeah I can totally see it, damn
that is a nice stutter
I have absolutely no idea.
It's not in my mod; it's entirely in game code.
The moment you bind any directory, even empty, you get this result.
Even without the mod.
rich quarry
The moment you bind any directory, even empty, you get this result.
So there's no way to fix it?
rain hedge
There's something I can try as an experiment
rich quarry
There's something I can try as an experiment
If you can find a way to even mitigate the issue it'd be nice, because right now it's very jarring
rain hedge
The fact it only shows up in this game is really strange
They all use the same CRI code for this
rich quarry
Yeah, I checked again with P4 and P5, I'm running them with no issues at all, which makes this weird
rain hedge
Technically it's possible this is caused by Denuvo and would magically resolve itself after a game update.
rich quarry
Technically it's possible this is caused by Denuvo and would magically resolve i...
But then don't all three games have Denuvo?
rain hedge
Yeah; thing is it inserts its own checks and virtual machines into random places throughout code.
Then developers playtest for a while to see if it's okay.
It could have hit the function that loads file from disk, or checks if it exists etc.
It's technically possible
There's no way for me to confirm other than waiting 24+ hours for it to disassemble
and comparing against another game by hand
am going to work on Virtual Filesystem :v
or procrastinate rather, as I did today
rich quarry
There's no way for me to confirm other than waiting 24+ hours for it to disassem...
Well, dammit
rain hedge
I'm trying something as an experiment for now though
WAIT WAIT WAIT
there is a way I can check this without waiting
not 100% reliable but,
lemme just download GamePass version
it uses different binary
I do not wanna make a CPK packer...
This reproduces on P4G64, making any directory bind at all slows down loading considerably
rich quarry
This reproduces on P4G64, making any directory bind at all slows down loading co...
Wait, it's an issue on P4G64 as well?
rich quarry
Seems like I'm staying on 32 for now (especially since 3/4 of the mods I have aren't ported yet)
Why's the new P5 the odd one out though (not that I'm complaining)
rain hedge
I'll try to rewrite the code to use another CRI api for binding stuff, see how it goes
I once failed; but maybe it'll work if I try again
for now I go to bed though
Note: If it doesn't work, it might work after I complete work on Virtual FileSystem
because that would allow game to discover files in game folder that aren't really there
rich quarry
Well, at least you have some ideas to work on, if you're still reading, I bid you goodnight
rain hedge
Checking my old notes, yeah, it should be possible;
Once I implement the VFS fully.
Maybe in a week or so that'll be done
Sorry for the inconvenience π π
rich quarry
Checking my old notes, yeah, it should be possible;
That's good, that's good
rich quarry
Sorry for the inconvenience π π
Don't worry, if anything, sorry to bother you with this
rain hedge
It's alright; it's a good test case for what I was already working on before I got here :v
ashen kite
Hello there. Can confirm this issue. My PC:Rtx3070, 5600x, 32gb of ram. I have both the hd music pack and colorful ui mods. I have horrible stutters in places with a bunch of npcs ive noticed. I lag very hard inside the classroom, the second floor, and the mall. I noticed when i went to the mall at nighttime with only a single npc, there were no stutters at all. Haven't tested combat like the poster has but I'm assuming I'll experience the same issue. Hopefully this can be resolved soon and a huge thank you to all you modders for your hard work. β€οΈ
Also forgot to mention that Ive experienced micro stutters even without the mods. When mods are on it makes the stutters worse so they become pretty noticeable.
wispy shard
I also got performance issues and just thought it was because my CPU is horrible, but because of the stutters here and there, the first scene with the evoker at the dorm rooftop has a huge audio delay.
Without mods the game runs perfectly fine.
buoyant flume
Can also confirm that theres longer load times with mods, especially after exiting a battle, didnt happen before. Currently using HD sfx, HD music, and p3f camera mods
gusty monolith
Is this for all mods?
wispy shard
Is this for all mods?
I would say yes.
rain hedge
Yes, it happens even if you make an empty mod.
rain hedge
This reproduces on P4G64, making any directory bind at all slows down loading co...
i.e.
gleaming sentinel
Has this issue been fixed?
wispy shard
Has this issue been fixed?
No
gleaming sentinel
I see. Do you know if it there will be an announcement somewhere when it does?
rain hedge
Maybe in a week or so that'll be done
Probably at least a week
gusty monolith
I would say yes.
Well crap
dawn scroll
I have the same issue. I also thought it was a problem with my computer or something, so I didn't bother with being the first to report it.
It ruins the oh so beautiful gameplay flow of barely having to deal with loading screens on an old game on newer hardware
feral shale
Yeah I get poor loading times as well
high girder
sorry to bump this - I think this might be a base game issue. Im running a brand new rig and its very easy to see the stutters in the field by opening the menu. Just watch "MAIN" scroll or any of the other animations and you can easily see the stutter. Its much less pronounced without mods, but it IS there. Tartarus menu is smooth AF, even with mods
specifically this is from the tartarus entry screen (VN mode) versus VN mode on the day akihiko takes you to the police station
gleaming sentinel
Idk, Im playing on my laptop and the game is running perfectly smooth for me, except for when I try mods
high girder
im running a similar rig to manzanita, maybe the rumors about nvidia drivers struggling could be it? I do have a weak internet connection, so maybe denuvo.....
wispy shard
im running a similar rig to manzanita, maybe the rumors about nvidia drivers str...
I have one of the best internet connection we can have in France and a really good PC (besides my CPU but I'm gonna upgrade next month) and it was smooth until I modded the game
Even if they are not ruining the experience because this is just a few stutters on some menus or cutscenes that can make audio delay
We will just have to wait for an improvement
ashen kite
im running a similar rig to manzanita, maybe the rumors about nvidia drivers str...
I have both a 3070 and rx 6800xt i switch out depending on the games I play. The stutters appear on both nvidia and radeon when mods are on. :"(
high girder
weird, Im running a 3080 and I get the stutters in non mod (very subtle in school, severe in mall). I will say I only noticed this after setting up reloaded, but might be occurring before. Probably something specific for me in that regards. Still, I'd probably use the menu screens "MAIN" scrolling text to check the severity of the slow down, its a lot clearer than moving the cursor around. Obviously all other games run flawlessly - I've seen some rumours that there is an update coming addressing amongst other things wrong with the port, audio and more. I think Luna is right - waiting on stability patches is the only fix
rain hedge
As I said it's something I'll be able to work around eventually.
Just been busy; and procrastinating a little
I'm gonna try being a bit more productive and rushing it a lil' I'm starting work in around 4-5 days
feral shale
No stress Sewer take your time
high girder
found the (my unmodded game) issue - frames dying horribly with the white speech mark animations playing above actor heads on the field. framerates fine outside of those field screens. Someone mentioned it in the steam community. Mods make the issue much worse as already known. Heres some footage - https://streamable.com/6w6e6g - this is something I can live with now I know its not the entire game, but maybe a mod that removed those animations might help those with weaker machines. hopefully that helps someone in the meanwhile
footage is modded since it helps pronounce the issue
gusty monolith
found the (my unmodded game) issue - frames dying horribly with the white speech...
Yep thatβs what Iβm experiencing too. And my laptop is able to run p5r at 60 fps a majority of the time.
ashen kite
weird, Im running a 3080 and I get the stutters in non mod (very subtle in schoo...
I get micro stutters base game and modded. The stutters are just way more noticeable and worse when mods are on. Ive noticed its anywhere with lots of npcs. During the day ill get stutters at the mall when its crowded. At night with only 1 npc or none i get no stutters.
wispy shard
This is what I think cause the stutters.
high girder
I realise my screen grab earlier didnt really show off what I was on about - modded runs fine for me except when one of these is on screen
the animation introduces micro stutters to the base game too
this is the "MAIN" animation I look at for finding micro stutters. You wouldnt really notice it in unmodded outside of animations like this
I get the BGM stuff might be causing slow downs for mods, but maybe the odd animation like this in the base game compounds the issue. unfortunately I dont have any game/mod dev experience (finance CRM dev, main transferable skill being ability to find cursed obtuse solutions) otherwise I'd try and look further to help find some way to tidy this up. My dumb monkey brain hates not knowing the full story
is this animation played in tartarus at all? I recall shadows having a similar animation in p4g but I havent tested as far as tartarus yet (outside of the entry field room)
high girder
alright, got into tartarus, average of 1 stutter outside of battle every 4 mins, doesnt require an enemy on screen. Stutters in battle limited to the all out attacks, almost identical to where p4g ate frames. Increased mods used (including new mod which replaces char models, pixels ability colour updates) and isnt really any worse by eye
for the sake of not spamming a thread that probably shouldnt have this info on it, to the modders here, is any of this useful, and should I post it elsewhere?
rich quarry
I actually think we need some kind of "Issue confirmed" or "Working on a fix" tag for those threads, especially since most people probably won't read all of this only to know if it's actually gonna be fixed or if it's an issue at all (which it is, in both cases)
rain hedge
The issue specifically is with binding directories using the CRI API (it will happen even with empty mod folders).
It's technically not intended to be used with shipped games; so I guess more of a debugging feature.
I can use BindFiles but that requires the files to be in the game directory which is no-no; hence I need finish with Virtual Filesystem first. Hopefully using that will fix the issue.
That will take a while; as it's not trivial and motivation/efficiency isn't 100% there for me right now.
ashen kite
I mean if we all donated and not complained that might motivate him. Still baffles my mind how yall do countless of hours of work with abysmal amount of donations.
abstract vortex
I mean if we all donated and not complained that might motivate him. Still baffl...
Don't donate to people as a motivator, donate because you appreciate them.
Ultimately we do this shit for fun. I don't personally care how much people donate to me - if I'm not feeling it, I'm not feeling it. I'll come back if/when I'm ready. This isn't my job and if it starts to feel like that, I'm even less motivated. We don't owe anyone anything.
gusty monolith
@ashen kite My now deleted response was towards me interpreting the issue being bigger than we initially thought not towards Sewer
dawn scroll
how's it going, sewer?
rain hedge
[I am still working on this; albeit pretty slowly]
Just had a lot of stuff happen lately in the past week:
gusty monolith
should show the led sometime lol
?
dawn scroll
how's it going, sewer?
rain hedge
I vibeth
vale wraith
gusty monolith
Huh?
dawn scroll
how's it going, sewer?
rain hedge
No; this is not dead.
Development is just slow.
I got the docs up for the part that will enable fixing the issue though :p
Then patching CriFsV2Hook will be trivial.
feral shale
Then patching CriFsV2Hook will be trivial.
This isn't small potatoes
rain hedge
Update: I can't release this because it has issues on Wine. A low level issue I can't work around while the code is in C#.
Have to port to Rust
... this gonna take a few weeks :/
feral shale
Fucking Linux
rain hedge
It's not Wine's fault; this would have blown up eventually in a future major runtime version.
grand marsh
One message removed from a suspended account.
rain hedge
I have to rewrite the while thing in native code
Including FileEmulationFramework too
So I just flipped over some tables and started working on Reloaded3 instead.
(it was all ready for release but could potentially have broke in next major runtime version)
That was 'okay for now', except it broke in Wine for same technical reason so I couldn't release it
vale wraith
:(
i'm waiting for the day sewer snaps and releases 'relowoded' on april 1st or something
only owo text and a very annoying theme
no i mean like no option to change it, but it has improved features over the previous installment so you have to use it lmao
relowoded
"the new and improved reloaded, but my sanity is gone"
pearl hinge
Guys hi! Just joined the server and I'm pretty sure that question was already asked, but... Does everyone get jitters when scrolling in the overworld? I finally bought the game to mod it and noticed this awfully chopped scroll. I mean, the game is already out for 4 months, just curious if all the people who modded the game had to bear with it.
digital ether
was noticing the same thing too, but seeing stuttering during overworld, wasn't ...
probably mean something like this, just navigating the overworld
pearl hinge
Yeah that's what I meant
As I understand from previous messages, this issue has been there since the beginning... It's curious because a lot of people praised the modded P3P on PC (like in Steam reviews / forums) but somehow I haven't heard many complaints about the laggy navigating. So I just wondered if everyone who modded the game got this issue.
rich quarry
Guys hi! Just joined the server and I'm pretty sure that question was already as...
It's worse if you mod the game, happens with loading screens and AOA too, from what I understood it's the game calling a bunch of stuff at the same time (and the issue is not present on consoles, I know because I own the game there as well) if you scroll a bit in this thread sewer already said he's working on it but it's gonna take more time than expected
Edit: in the overworld is mostly an issue with the "talking" particle (those three white things above sprites, idk what to call them)
pearl hinge
Thanks for clarifying! Will be patiently waiting then. Really wanted to play through P3P, but it simply isn't that enjoyable without cutscenes and uncompressed BGM 
rain hedge
Update: I can't release this because it has issues on Wine. A low level issue I ...
It's gonna take a long time π
indigo reef
Guys hi! Just joined the server and I'm pretty sure that question was already as...
I started experiencing this + other things when I switched to a new pc a couple of days ago. I installed the exact same mods I had on my previous pc and stuff got messed up, I get the jittery navigation you mention, stutters on all out attacks, random like half a second freezes while navigating Tartarus and two second freezes when Mitsuru uses Bufudyne
I thought the issue was the mod load order so I started reading the mod pages to see if the order I was using was wrong and messing around and the issues were there even with one just mod enabled
What's curious to me is that on my previous pc everything worked flawlessly, no jitters, no stutters, no anything, like I said running the exact same mods. For reference in case it's of any use for debugging or something previous pc had a 10th gen i7 and a 2060 and current one has a 12th gen i7 and a 3070
pearl hinge
Now that's curious... 
gilded wave
this is a thought that crossed my mind. if the issue is with reloaded. could we theoretically just rebuild the cpk with our mods installed into it? (i'm very amateur at modding and don't totally understand how files for games work and stuff)
abstract vortex
It's possible but very very very inconvenient
gilded wave
It's possible but very very very inconvenient
are there any cpk assembly tools so i can do this for personal use?
gilded wave
Well i think i found one but the thing is i donβt think it can read the files that are in japanese
abstract vortex
#1045463162452987924
gilded wave
Thank
devout thistleBOT
Seems like this thread is still kinda active and hasn't been closed but yeah definitely experiencing this in p4g for sure with a 9700k/3070. Tried swapping mod folder/game folder to ssd but it seems like it doesn't really change anything between hdds or ssds.
As was mentioned here it's actually fine with persona essentials and inaba exe patcher left enabled it's just further mods that drastically affect load times. Haven't come across any stuttering but that might be more related to P3P.
rain hedge
It's not dead, it's never dead. Just that the amount of work to get there requires overhauling other things down the stack, and porting some huge projects in their entirety to other programming language(s).
It'll just take a long time. Nothing more, nothing else.
abstract vortex
I think I'll just close this thread. I was hoping it being open would prevent people asking the same questions again and again, but I guess not.
rain hedge
I can use `BindFiles` but that requires the files to be in the game directory wh...
While I still work on porting stuff to Rust so I can port Virtual FileSystem to it; which would ultimately make fixing this properly possible...
I found a temporary workaround; while discussing with a guy who wanted to mod a 32-bit One Piece game. I had a dumb idea, and it worked.
It'll modify/pollute your game folder [Note: it will NOT edit any existing files], so it's not an ideal solution; but it's better than waiting till the end of the year at this rate.
abstract vortex
I'm listening 
rain hedge
I just had a dumb idea to symlink every single subfolder of the bind folder into the game directory. I have no idea how I didn't think of that before. Maybe I did; but decided to go VFS then forgot π
Surprisingly, it works.
There's only 1 catch. The current approach with BindDirectory is CASE INSENSITIVE.
The new approach is CASE SENSITIVE.
I'll need to write some logic that will look up the original CPK files to fix casing. Thankfully there's already a fast lookup into existing CPKs; courtesy of the tools I built for file merging.
rain hedge
I'll need to write some logic that will look up the original CPK files to fix ca...
If there are any mods which insert files that are NOT in any of the original CPKs, they may break!
I'm not sure if there will be any automatic way to fix them; at the moment.
[i.e. This will require testing]
I pulled an all nighter, so I gotta nap, then do work stuff for today.
I'll clean up whatever I have after, see where it goes.
oh yeah... and I installed Windows this morning just for this π
(it's easier to debug natively)
[I've been Linux only for past 2 months now]
rain hedge
If there are any mods which insert files that are NOT in any of the original CPK...
There's also another caveat: You can't run multiple instances of the game with different mods enabled.
Because symlinks are made into the actual game directory, modifying the directory. When the Rust ports are done and VFS (Virtual FileSystem) is available down the road, that shouldn't be an issue though.
clever forge
There's only 1 catch. The current approach with `BindDirectory` is CASE INSENSIT...
you say this will break new/non existing files because things are case sensitive, but despite this, in, say, P5R, the game code will always request files in lowercase, while the files themselves are all uppercase, surely something within the game itself is resolving this conflict
rain hedge
you say this will break new/non existing files because things are case sensitive...
I only tried mods which replace character models so far. The replacement only worked when the case matched exactly what was in the original CPK file.
clever forge
tha'ts very very odd
rain hedge
If either any of the folders' cases or file case didn't match, it would fail
You could use the slashes interchangeably though, at least
clever forge
here's an example
this is from P5R, game has a massive list of models/animation files to preload during boot
all lowercase strings in exe
rain hedge
That's hella weird
clever forge
so surely something is going on somewhere then
clever forge
very very odd
clever forge
maybe the crifs functions are forcing all uppercase somewhere before loading the file?
rain hedge
It might be possible, I haven't tested with p3p/p4g
The other 2 games have files lowercase in CPKs, so lemme find a mod I can quickly test with
rain hedge
p4g just commits seppuku after a certain file count π
Guess I need to increase file limit
rain hedge
It took me a while to find the right file in p4g [whoa, the amount of duplicated textures...]
Yeah, seems it's case sensitive
In p4g only lower case works [matching CPK file]
rain hedge
It took me a while to find the right file in p4g [whoa, the amount of duplicated...
[because I was lazy to increase file limit :v]
[I'll need to hook into another function to increase limit of open files]
I'll just do whatever is current count + Max(numBindFiles, 65536), should work/
Since I can't predict when people will add files with Hot Reload, or how many of them
or if they'll load extra mods at runtime while game is running
clever forge
its not really useful on the new ports because of how they load files, P5R notoriously keeps files cached for a while
so if you load a persona model and exit the menu, the model is still cached so that when you enter the menu again the model is instantly there
clever forge
clever forge
The arcana is the means by which all is revealed.
rain hedge
By the way; I do have it 'ready'; though I'm not satisfied. Startup is now too slow.
So I must keep investigating.
rain hedge
I'm gonna hook lower level; and remove all hardlink/symlink stuff entirely.
Then implement caching for file info.
Should improve boots; however, will take some time to code, as I'm back to work today haha.
[Also will make it harder for wider game support aaaaaaa]
clever forge
I dont know how useful this info is but the Switch custom firmware Atmosphere iirc does file caching for modded files?
could always be worth looking at that code
rain hedge
The overhead is in calling CreateFile for 5000+ files
When you have a huge mod list.
When you call BindFiles, it opens all files at once.
And then queries their size.
Thing is, I can go back to BindDirectory and hook internals so performance doesn't suck; but then, not all games support BindDirectory
Difficult to build 'one size fits all' solution
clever forge
Atmosphere does that too lol, it queries all the game files and makes a cache of some sort from what I understand
and caching all filepaths was causing OoM issues with some games that had ridiculous file count
(like zelda totk)
rain hedge
Though I am curious why it is so slow in the first place; I remember writing a benchmark at one point, I could open 20000 files in 1.5s
Well; I'll hook into CRI internals and kill symlinks/hardlinks altogether, see how things go.
clever forge
if you're curious https://github.com/Atmosphere-NX/Atmosphere/
rain hedge
Dajiobu π
I'll figure something out
Worst case scenario I'll figure out some hybrid approach, where some games do X and others do Y
I do wish Reloaded3 was here tho; I'd prefer to write Middleware hooking code in Rust
As I'd like to do Switch sometime too
The problem is somewhere in CRI
There's gaps between when they try to load next files
i.e. it's not scheduling hard enough
Oh I fixed it; from 50 seconds to 642ms
neato
Still am gonna be hooking internals. Currently I query CPKs to get correct casing for file names in mods; but I found out I can't rely on that. Some games have them in another folder than the binary entirely
rain hedge
Oh I fixed it; from 50 seconds to 642ms
Still not ideal, but better
rain hedge
I'm glad though that I could get something 'acceptable' out that doesn't involve writing full VFS.
Since I want to write VFS in Rust; I don't want to wrap it in C# just to backport to R2
I'd rather write it straight for R3
rain hedge
An experimental update for CriFsV2Hook, which should massively improve load times (main source of stutter).
Not thoroughly tested (with emulators/merging); so am putting it out there.
Added: Feature to log all file reads from CRI.
Added: Feature to log all replaced files as they are replaced live at runtime.
Improved: Startup time by around 2x.
Overhauled: Overall hooking approach to make it portable to other games.
Changed: Removed use of hardlinks entirely, Bind folder is now empty.
Added: Support for games which use UTF-8 file names.
π
For more details, see: https://sewer56.dev/CriFs.V2.Hook.ReloadedII/
I added support for every single other CRI game I own.
Fully supported but you can't do anything useful with them:
These games 100% work, but the games don't use CPK files with file names to load their content; rendering the mod effectively useless for them.
Recognized but unsupported games:
For these games, the needed code is not present in the final binary/exe, so can't be fully supported with current approach. I'll support them in Reloaded3 (2024) if I ever make a CPK emulator.
rain hedge
I patched that, yes
Was it not case insensitive previously? (so I can add to changelog)
I did make it case insensitive in this update.
rain hedge
I mean the version that people were using up until now.
I did fix case-sensitivity, so assuming it was previously already case-insensitive, there's no difference there.
torn zodiac
yeah
just checked on p5r with a morbillion files (the entire model directory replaced, plus some other mods)
actual ingame load times are instant now
startup time though
clever forge
startup time though
issue of skill im afraid
torn zodiac
perhaps
clever forge
my game loads lightning fast now, didnt see some of the usual stutters
and my bgm doesnt work now lmao, the eternal struggle
torn zodiac
(less files this time)
i guess it's denuvo trolling
smt3hd takes 20 billion years to boot
sonic frontiers is about the same as p5r (without mods)
clever forge
mine boots instantly
clever forge
i remember asking sewer this a while ago
i dont remember the answer i got
because mod cpk would pretty much solve all bgm issues lol
torn zodiac
shoving all non-mergeable files into cpks would probably help
upbeat lake
awb emulator doesn't seem to be working for me anymore either with this update, although the tracks look like they're being injected in the log
feral shale
smt3hd takes 20 billion years to boot
You're still the only person I've heard of that has issues with Nocturne taking forever to boot
rain hedge
Depends if there's stuff to fix or not
torn zodiac
You're still the only person I've heard of that has issues with Nocturne taking ...
doesn't work on my machineβ’οΈ
rain hedge
(also yaaawn I just woke up)
rain hedge
shoving all non-mergeable files into cpks would probably help
The issue is when those files become suddenly mergeable
And we have to extract them suddenly and we have a clusterfuck going on.
torn zodiac
how would one merge gmd
rain hedge
The concern is moreso on end users putting mergeable files in CPKs, not knowing they are or could potentially be mergeable.
It also fucks with priority since manually bound files will always take priority over CPKs, which goes against the mod load order
I can (probably) fix that actually, but it still creates complications for merging.
rain hedge
In this case, end users are mod creators.
torn zodiac
ah 
rain hedge
It's not an unresolvable issue, it's just something I don't want to invest time in right now
torn zodiac
understandable
rain hedge
When it comes to boot times:
It's proportional to number of bound files, and depends on if these files were recently accessed.
CRI opens all loose files ahead of time during boot, to make it so they can be accessed more instantaneously when needed at runtime.
When you boot for the first time with many files, ya, expect longer startups. If you're a mod creator though and are rebooting constantly, or are just fiddling with your mod load order, because the files were recently loaded, they should load again faster.
I get around 500ms on Adachi over everyone (5000 files), on repeat boots. Not ideal, but it's there.
torn zodiac
yeah i noticed it's faster on repeat boots
rain hedge
CPK binding would reduce this much closer to 0 but it's not just something I want to invest in right now. After noticing I can do this approach to binding, and that I could fix stutters, I actually abandoned another project (in the middle of it), for 2 weeks just to get where we are now.
torn zodiac
π
rain hedge
invest in right now.
Maybe if I make an emulator for it down the road, then I could have 3 tiers of support.
Tier 2 would make it work with any past, present and future game; without any explicit code for it (but you can't hot reload or edit files at runtime).
Tier 1 would work with any CRI game that uses a CPK. (e.g. NieR), but can't bind files. It could load/unload mods at runtime, and hot reload (if you replace entire CPK)
Tier 0 would be existing solution which works for most but not all games.
Development of any emulator is entirely out of the question until Reloaded3 (middle of 2024 thereabout), because any emulator written in C# will eventually most likely break due to GC Transition Shenanigans; so all existing ones need ported to Rust already as-is.
I could do Tier 1 CPK binding today though; if there's really demand for it, I have no issues giving it a go. I'll just need some assistance in ensuring merging etc. can accommodate for this; because this would introduce breaking changes.
vale wraith
holy shit reloaded 3 real???
rain hedge
By breaking changes I mean: Now we have a situation where some mergeable files are CPK sourced and some are file sourced, the current API we pass to mods only have file paths, so all users of these APIs must be updated to account for the new requirement.
With CPK emu on the other hand, all of the existing logic could be kept, and instead you could make a virtual CPK that Tier1 could use instead of using real ones.
Hence why I'd rather wait for emu
[which I cannot do until R3]
It won't improve startup, to same degree real CPKs would, but it would be a few times faster than current. Since instead of opening each file, you're just doing a file search, which returns details of multiple files at once.
=========
On the other hand am surprised nobody started meming on the fact I added CPK modding support to games that don't use CPKs
wind fable
This is very cool, nice work.
I did have to clear Persona Essential's cache for it to work though (it'd immediately crash on boot of I didn't). Not sure if that's just an issue with me or an actual issue though
rain hedge
Apparently emulators are borked; I presume it's because I'm no longer using bind folder
And some hooks require file paths since handle isn't provided
So I have to look into that
wind fable
I could do Tier 1 CPK binding today though; if there's really demand for it, I h...
imo what you've done now is a huge improvement. I think we'll be fine with it for now
clever forge
yeah any music mod is completely dead now
but only music mods, other mods that use acb/awb (like voice mods) work fine somehow
(at least for p5r)
rain hedge
I was just so tired that I released it as is π
[after already delaying it a few days and working on it daily for 2 weeks]
wind fable
Oh, actually I have to clear the cache every time, I guess something's breaking when loading stuff from it 
Guess I won't be using this for now
clever forge
oh, that's odd, i didnt have to clear cache for it to work, i dont think
rain hedge
Oh, actually I have to clear the cache every time, I guess something's breaking ...
Look into it then
While I check AWB emu (once I finish eating here)
wind fable
Fair, I could do that
Oh, it's happening when dumping the bf files. The path is completely messed up
clever forge
oh yeah shrinefox mentioned bf merging wasnt working
rain hedge
And some hooks require file paths since handle isn't provided
I imagine it might be this
wind fable
Hmm, DumpFile wasn't being called at all with the previous version (which makes sense, if it's using a cached version I'm fairly sure it'd never get near there)
I have to go for a while but when I get back in a few hours I'll look into it more if nothing's been worked out before then
rain hedge
Aight
wind fable
Well I found one problem, I was passing the filepath instead of the route to one of the functions. But now that I've fixed that I've found that the route it gets from FileAccessServer is null for some reason 
rain hedge
oop
Meanwhile I just got back after trying to take care of myself for once (with some exercise), am looking at the AWB stuff
well, just got started
AWB Emu itself is fine obviously; just the CPK integration is borked
These low level interactions should probably be better documented come emulator Rust port; so unexpected breaking changes don't ever happen again
wind fable
This is confusing but I somewhat understand it now. So, when FileAccessServer passes a route to an emulator it is seemingly always just the full path, not a route as I thought it'd be (like init_free.bin/field/script/field.bf). I can't tell if that's intentional or not, seems like it sort of is
That's what's causing the crash when dumping bfs, I'm not really sure how to fix that but for now I'll just turn of dumping them. Now that I've done that I found there are more problems, lose bfs don't compile now π«
wind fable
Oh, I didn't know about that
rain hedge
I added this in this version
[it's actually in changelog above]
I wish
he just kinda stares at me all the time
it kinda spooky
I think I might know what's up in the AWB case
Older version of CRIFsV2Hook code would scan the Bind folder for files to inject.
This one doesn't; it uses the binder context which was previously used to create hardlinks directly.
Converts them to relative paths, binds them with CRI and later converts them back to full paths in an internal hook.
I think I just need to register AWB into bind.RelativePathToFileMap in the AWB case.
If that's the culprit, I could make a patch that will auto-register whatever it can find in 'Bind' folder.
And that should be good.
wind fable
I'm pretty sure it's something similar with the bfs, I just put the cached ones into the bind folder before but that doesn't work anymore
rain hedge
Ignoring files in Bind is a breaking change either way; that I missed without noticing
wind fable
Yeah, that did fix it
rain hedge
So even if it's not the culprit, I'll fix it either way
Well, didn't fix it for me (might be to do with AWB+ACB), but the functionality needs restoring anyway, so I'll see
wind fable
Nvm, it fixed the error but it doesn't actually work. The files don't get replaced π
I'll just wait for now and hope whatever you do happens to fix it for bf emulator as well π
rain hedge
oki
rain hedge
@wind fable AWB/ACB still borked, but can you build latest commit and see what happens for other emus?
It's weird, everything ok in logs, but I'm clearly missing something
wind fable
I'll check now
They're still not working
Well bf emulator, pak emulator seems fine
And to be clear it's only with loose bf files that I have to register with crifs redirector that stuff breaks
Oh, it was actually my mistake. I put the wrong path in the binder input, I set it to the right one and it works fine now
Or not, it's only working for some files π€¨
Oh, is stuff case sensitive now? I think that might be my problem
Yep that was it
I love how they decided to capitalise event bfs but leave almost everything else lowercase π
rain hedge
fff
rain hedge
Yep that was it
I patched the CRI methods for finding and opening files to use the same case as the bound files
(See: CriFsBinderFindImpl in CpkBinder)
Is that not holding up?
wind fable
I guess not
I had a file e845_500a.bf and it didn't work, renamed it to E845_500A.bf and it did
rain hedge
wtf weird
Can you debug that?
The ACB situation is weird too; everything seems to be working as intended according to logs but I get... well, not good
wind fable
Sure, I'll try in a bit
rain hedge
It's really weird.
rain hedge
Like, I dump emulated files and they're okay
And the game opens the handle to the file when I bind it
FileEmulationFramework reports overrides
and yes... audio is ded ingame
wind fable
I don't understand, stepping through the code everything seems fine, it finds originalKey as the lower case one, err is 0, exist isn't null but in game it's definitely not being replaced
rain hedge
Yeah; that's what I'm experiencing :(
In P5R running a sound mod kills the game entirely even, as if ACB wasn't patched; but according to logs it is
wind fable
So how does it actually work? Whene you call findFileHook with the tempStr does that load the lowercase version into where it expects this to put it?
rain hedge
It's supposed to replace the string in the Find operation such that it matches the string used to bind the file (with BindFiles) exactly.
hmm, I renamed a model mod to use upper case, now it doesn't load into level in P4G
Maybe my general approach for case-insensitive is borked
Or maybe I changed something along the way that mad eit no longer correct
wind fable
Fwiw it makes sense to me, seems like it should work...
rain hedge
Lemme just return true if the file is found in Find function.
Because we know it exists because we bound it
wind fable
Maybe it's doing its own search in a case-insensitive way but just prioritising the first thing it sees?
So it sees the original one first and returns that but if we use exactly the same the original gets replaced? Just a guess
Because as I think others have pointed out the games do sometimes use different case in their loading code than what the files actually are
rain hedge
Because we know it exists because we bound it
oh nvm I can't do that, since need to return handle info
rain hedge
Because as I think others have pointed out the games do sometimes use different ...
Yeah, funny stuff happens
P4G tries to open files starting with backslash (lol)
but only for some files
P5R tries to open files using both path separators despite CRI normalizing it to /
(it's noted in CRI docs)
I have correct case and redirect Find == OK
I have wrong case and redirect Find == Deadlock
I have correct case and don't redirect Find == Nothing
wind fable
Wrong case and redirecting gives you a deadlock?
rain hedge
Yeah seems like it
(hair)
Weirdly, if I don't redirect find, it doesn't; work, despite path not being changed
nevermind it works
hm
wind fable
Is that with the current commit or local changes? Using the current one (at least with this bf) it doesn't deadlock, it just uses the original
wind fable
Ah ok
rain hedge
Well; I didn't edit any logic
except for adding a log message in Find
Just seems like my case fixing in Find doesn't work
During Bind:
When loading file later:
Stuck loading:
I guess I only tested on P5R when I initially implemented this logic
I wonder why it deadlocks
wind fable
Is the path stored somewhere in what's returned?
rain hedge
hm,
you have a point
One of the out params for find is CriFsBinderFileInfoTag
And one of the fields there is path
wind fable
Maybe it does a double check after it's done...
rain hedge
I could try replacing that with the original passed to the method
P5R doesn't have the deadlock for reference
I guess I'll change the path in the returned handle to whatever the game passes to the function
See what happens
Though that'd be a hack, not a solution, and lifetime of returned string couldn't be correctly controlled
Unless I only deallocate with criFs_FinalizeLibrary and reuse strings; that would work
I'll assume it's a case of the game checking the returned file path
Replace it, and see what that gets me
rain hedge
I started replacing returned paths, no difference
wind fable
π¦
rain hedge
I guess I'll hook the file loader instead
And do the case fixup there
This will be annoying though as I'll need to update sigs for every game (> 10 games).
Hmm, there is also an alternate solution
Parsing every CPK as game opens it
Short break then I'll try it
rain hedge
[no progress since yesterday because I wasn't feeling well]
upbeat lake
No worries
rain hedge
(Still no updates, I've been having a headache this whole time. Am just resting in bed)
I was either sick or have an ear infection, I don't know which yet.
Anyway what I wanted to do was hook internal function load_registered_file instead of Find, to fix the casing problem.
I don't know when I'll get to test it, for now I just rest
leaden mauve
damn man, hope you heal quicklyπ
rain hedge
Anyway what I wanted to do was hook internal function `load_registered_file` ins...
Seems like this worked for P4G/P3P, and restored AWB emulator functionality too
Though something weird with P5R, they use a different binder handle for every file load; I have no idea what's going on there
rain hedge
Back from walk and some exercise
Anyway, I figured out what's up with P5R
They do something funny where they bind a file inside CPK despite already binding CPK
They have custom logic for loading encrypted assets like character models
rain hedge
oh right PS4 did some shit like that
Yeah, and that carried on to PC
My implementation of decrypt is actually a few times faster than what the game itself actually uses; could replace the game's implementation with my own if I feel like it
clever forge
rain hedge
It's only 0x400 bytes of XOR though LOL
clever forge
yeah i remember reading it from lipsum back during the ps4 days
if yours is faster and you can get away with replacing it, id say go for it
rain hedge
Yeah replacing it is trivial
rain hedge
@wind fable Can you test main branch.
I fixed case sensitivity (at least AWB emulator seems to work)
wind fable
It seems to work fine now with bf emulator
rain hedge
hmmm
wind fable
I do still need to fix bf emulator crashing when dumping files though
I change my mind, something's going horribly wrong when it loads cached files still (works fine on the first launch)
Looks like the procedure positions are getting messed up or something like that 
wind fable
nvm, using my version of Persona Essentials that registers the bf files to the binder input fixed that. Seems it's actually all good now
I've made a pr with my fix, hopefully everything else with this is working now π€
rain hedge
Alright : π
wind fable
Bad news, P3P seems to get stuck booting with this new version (gets to bind complete, registers sysdat\umd2.cpk then nothing)
Better not release anything for now, I'll have a look tomorrow if nothing's worked out by then
Good thing I actually launched P3P, that would've been funny if the game that started this was broken with it π
rain hedge
Alright.
I booted P3P earlier, had no issues, but didn';t really try any mods aside from a simple one
so no merging etc.
wind fable
I think it's a problem with emulators, it was late when I found this out so I didn't thoroughly checked but it seemed like it was if anything used PAK or BF Emulator.
Will look properly today
wind fable
I've found the problem, the game's stuck in an infinite looping waiting for the bind status to become complete.
Not sure why it isn't being completed though, based on the logs it's done
It's waiting on the binder with id 10 but it doesn't look like crifs hook ever does anything with an id of 10 
rain hedge
I'll leave it to you to mess around with; since there's no issues on my end.
If in doubt, doublecheck the hooks tho; I am hooking BindFile(s) now too
wind fable
Oh, I think I know why it's breaking (but not a solution yet).
sysdat/umd2.cpk uses / for the separators in its files whereas everything else uses \. Adding a condition to just ignore that file makes everything work (obviously not a real solution)
nvm, others use a mix of both it seems. Oh well, at least I know it's something to do with that cpk
wind fable
I honestly still have no idea what I'm looking for, all I know is stopping BindAll from running on sysdat/umd2.cpk (in BindCpkImpl) fixes the problem and the problem is GetStatus is returning error forever π
rain hedge
I honestly still have no idea what I'm looking for, all I know is stopping `Bind...
I just woke up, what status does it return?
You could mess around with these; https://github.com/Sewer56/CriFs.V2.Hook.ReloadedII/blob/c9568fce3dfffa4ea58b6fcabb3e78828c15872c/CriFs.V2.Hook/Hooks/CpkBinder.cs#L151
When you hit limits, it won't always report to stdout ; but it's possible we've hit one.
wind fable
It returns 6 (ERROR according to the enum in your code)
wind fable
You could mess around with these; https://github.com/Sewer56/CriFs.V2.Hook.Reloa...
Oh, that was it it seems. I changed maxFiles to add 4 * the count instead of 2* and it worked
It seems like you're binding every file to every cpk, if that's the case should it be _content.Count * numCpks? (Idk how you'd get that number if it is)
rain hedge
It seems like you're binding every file to every cpk, if that's the case should ...
You can't really predict it
Unfortunately
rain hedge
More specifically, we add files to each binder.
Usually games bind multiple CPKs under 1 binder, but P3P uses 1 binder per CPK
Not sure why, might be some 2 disc leftover from PSP
rain hedge
Can we just make it a really big number (int max or something)?
We can, though it'd eat a lot of memory haha.
wind fable
Oh, ok
wind fable
Does it have to be set at the start or could we change it later on?
rain hedge
At the start
wind fable
rip
rain hedge
Anyway, here's an idea.
We hook BindFile(s) already
We can extract current max file count (at init time), keep track of it [inside the hook], and throw a warning if exceeded. (CRI does have its own warning but doesn't throw under all contexts)
Though for now we can probably get away with _additionalFiles = _content.Count * 6; instead of *2
Though this might eat up memory if e.g. someone uses Adachi over everyone mod.
wind fable
We can extract current max file count (at init time), keep track of it [inside t...
How would that actually solve the problem though? Wouldn't that just be warning us before it freezes?
rain hedge
Yeah
rain hedge
Though for now we can probably get away with `_additionalFiles = _content.Count ...
IIRC P3P uses 3 binders so this would maintain the intended current behaviour for P3P
But it's not ideal.
So either we provide some export/API to override expected binder count (and multiply that math by given binder count). i.e. 3 binders means we do *6 instead of *2
And/or just straight up increase it, but that will eat up RAM in huge mod setups
wind fable
I don't think a couple of megabyes should really matter tbh. I get that it's not ideal but at least for now I think it's fine
Isn't this whole thing a less than ideal solution anyway? What's the harm in adding a little more jank to it π
rain hedge
They could for 32-bit games that are starved on address space.
wind fable
Well, none of the games I care about are 32-bit lol
idk, up to you how you go about it
rain hedge
We can extract current max file count (at init time), keep track of it [inside t...
And do the counting in debug builds only, so this issue can be caught in future games
Though P3P is only game I've seen that uses multiple binders
wind fable
That sounds like a good idea. So I guess we'd just add something to Persona Essentials that'd use that API when the game's P3P
rain hedge
Yeah
Gonna take a quick walk and I'll look at it when am back.
Otherwise feel free to PR it
Either works
wind fable
Well since it seems like you didn't end up doing that I just made a pr with it.
I'll make one with the change to Persona Essentials if you want after the CriFsHook one's been merged so I can update the submodule
rain hedge
Yeah, all's good
rain hedge
Do we just roll it out to end users? I don't know how thoroughly tested stuff was.
Would just need to prep it so all repos get updates at once
wind fable
I wouldn't call it thoroughly tested but I've played P3P and P4G a fair bit with it and haven't found any issues apart from the ones that we've fixed
I don't think there's really any chance of having it "thoroughly tested" tbh. I say just roll it out and pray nothing explodes
rain hedge
Well, here goes nothing.
Just let me get the releases prepared
Okay time to cook and hope the world doesn't burn down
Gotta get the API call into Persona Essentials too before release
YOLO
@wind fable Should be live.
rain hedge
# About
An experimental update for `CriFsV2Hook`, which should massively improv...
This changelog is still valid, nothing really else to add there
Throw an announcement, hopefully we'll be able to catch any issues in a relatively short time frame if there's any.
wind fable
Yeah, I'll do that in a minute
frozen jungle
just pitching in to congratulate and celebrate π 
this alongside having to open and close the console until the game opens have been plaguing my mod testing/debugging sessions 
clever forge
well, i briefly checked P5R for a few mins and at a surface level everything seems to work fine
even BGM which was dead in the first test version
subtle agate
briefly went and checked P4G and any event seems to crash the game with nothing relevant in the console (if there's anywhere else I should be looking for logs let me know)
currently checking my mod list just to make sure it isn't something else, but was not having these issues prior to the update
wind fable
π
subtle agate
I loaded it with only persona essentials, didn't seem to crash this time, might be a different mod. maybe false alarm haha
rain hedge
I haven't got any playtime beyond basic tests myself, I do have a bunch of saves though, up to end of Yukiko's castle anyway
subtle agate
is there something that changed with this update with regards to file structure/naming/etc?
rain hedge
The entire system for loading files under the hood was reworked
It's more complex, with more edge cases, but also way faster and more flexible game support wise
(There's 10+ games that use CPKs supported now)
subtle agate
gotcha. ive been out of the loop with all this honestly haha.
in theory should all previous mods work without issue?
wind fable
In terms of how a mod actually structures its files it should be the same though
In theory, yeah
subtle agate
alright. yeah i'm troubleshooting right now to see if it's any in-development mods causing these issues or something else, i'll get back to you all once I find out
rain hedge
BTW, on a fun note. I have a method which would make load times for certain modded file types faster 
than even the vanilla game. [And it'd work on other games too]
But that'll have to wait till 2024
subtle agate
thanks for all your hard work btw guys. gonna have to send you all a donation when i can, you guys have been putting out some awesome work
rain hedge
rain hedge
But that'll have to wait till 2024 <:meow3c:797678999609802752>
It's emulating regular files from archives, much like the emulators create virtual files that don't really exist.
So you know, instead of emulating an archive based from files, you can also do the reverse.
π
subtle agate
alright, turns out it was a development version of a mod i have installed. false alarm, sorry about that lol
although that mod was working prior to this update and I genuinely have no idea why it isn't right now, so if you have any insight regarding that...
rain hedge
Throw it over to Swine, in case it's an emulator issue.
subtle agate
sounds good
rain hedge
Seems like it broke GamePass π (P5R)
The sigs are matching as P3P/P4G, probably finding wrong function somewhere
devout thistleBOT
bold walrus
