Add bin merging (+ any other major Aemulus features still missing) to Reloaded II / PersonEssentials

[mortal yarrow]

like flowscript merging into a tbl merge?

or would one take precedent over the other

[scenic sand]

wdym, like recursive resolution?

yeah thats a thing

[mortal yarrow]

oh really?

ok good to know

[scenic sand]

(incidentally im apparently having trouble with that over here)

tangentially, @true bronze @lethal orbit did either of you (or anyone you know whos done code mods) get this window when building bc i havent seen it once

[lethal orbit]

nope

[true bronze]

You need to add a Debugger.Launch();

Iirc at one point the template had it so if you built as debug it added it but that was reemoved at some point so that page is wrong (or at least I don't have it in my version)

[scenic sand]

ah

[true bronze]

If you wanted it whenever building as debug you'd just add

#if DEBUG
Debugger.Launch();
#endif

[scenic sand]

do i just chuck that in mod.cs

[true bronze]

Yeah, just somewhere when the mod's starting up (or whenever you need it to launch)

[lethal orbit]

@cedar shuttle made prs for spd merging

[cedar shuttle]

I'll have a look after work stuff :p

[cedar shuttle]

I'm gonna cry when I'm gonna have to walk around to convince everyone to port Emulators to Rust

😭

It's gonna be real hard, haha. Even though, it's urgent, aaaaa

[scenic sand]

excuse for me to learn a new language

[lethal orbit]

Take your time I still need to make the API actually useful lmao

[cedar shuttle]

😅

[lethal orbit]

Finished it last night

[cedar shuttle]

Cool trick with the ID notations

[cedar shuttle]

I fell asleep earlier but I started looking at this 1 hour ago :p

[cedar shuttle]

Spent 2h in already on the PR review side. I need to nap, but I should have feedback ready later today.

@true bronze and since you coincidentally have BF emulator open (lol)

Interfaces packages shouldn't reference any external dependencies, seems I missed that when reviewing BF Emulator.

Because if they do have them, mods that reference that interfaces package will be locked into same dependency versions as interfaces.

[true bronze]

Oh, I see. Idk why it even referenced that, it didn't use it at all it seems

[cedar shuttle]

(I noticed BF interfaces had this problem when reviewing the SPD code, since the SPD boilerplate code was copied from BF)

[true bronze]

I have a bug fix for an encoding thing with bf emulator so I'll make a pr with both soon

[cedar shuttle]

Aight

[cedar shuttle]

Nothing too groundbreaking btw, bunch of small fixes/changes so far. And minor performance improvements.

For instance I found a bug where a stream doesn't get disposed if a certain condition is false.

(And minor code style comments here and there)

[lethal orbit]

I'm also taking feedback on usage docs 👀

[cedar shuttle]

One of the feedback pieces I left before napping was that the section on Id(s) doesn't mention the negate/exclude selector.

But the sections above did.

(also aaaa I wanna go sleep again, but I've been bamboozled into a hairdresser trip)

[cedar shuttle]

Damn, what a long review process

[lethal orbit]

In the docs? I put it in the texture replacement section because that's the only thing that can use it

[cedar shuttle]

ah, I see.

Yeah, I saw that in the code.

Funnily enough I just found a bug that affects like half the emulators while reviewing this :V

Because of copy-pasted code from AWB Emu

Okay, review for the Emu is done.

After like 4-5 hours spent total I think

Now for Essentials Side

It's just a bunch of tiny performance and code style changes on Emulator side.

Biggest change needed is making the Emulator a stream based one, rather than keeping the new file entirely in RAM; as that is inefficient for textures. There's not that much work needed to achieve that thankfully. Just sourcing raw texture data from file slices instead of reading it in and using MemoryStream.

[lethal orbit]

Thanks for the feedback! I've read half of it, gonna read the other half later 🫠

[cedar shuttle]

It's a-ok :p

[cedar shuttle]

[lethal orbit]

[cedar shuttle]

I'm depressed today, so sorry if I'm late to respond to things

f

[lethal orbit]

no worries! Hope you feel better soon

[empty galleon]

hope you feel better! depression is hard

[true bronze]

The other day I came up with an (in hindsight pretty obvious) idea on dealing with updating script compiler libraries without breaking old mods. I've added "Aliases" to script compiler so any past names of each function should still be detected.
With this we (probably meaning me ) should be able to safely document new functions as well as rename old ones (particularly P4G) so they're consistent between the games.

I'll send a build in a bit in case that sounds interesting to anyone. Either way, I'll add it to my existing pr once I've done a bit of actual testing

Also fwiw this doesn't use anything specific to bf emulator so it'd work with Aemulus for anyone that cared about that (not me at this point)

[scenic sand]

ooo

[mortal yarrow]

that is so handy oml

thank you swine

instant upgrade

[dusk crow]

LETS GOOOOOO

(as someone who's documented/renamed more than a few flowscript library functions in p3p myself locally it'd def be handy to be able to get them added without causing most existing merging mods to break lmao)

though i still would want to actually double-check and verify the functions do what i think they do before attempting to get any of my library changes merged lol

might even be a little handy for library overrides in BF emulator too since people could rename functions for their mod and get ahead of library changes if they were able to add aliases in their library override, which would prevent the issue of the renamed functions causing other mods to break if they used the same functions but with the original/default names

(though that would get a bit funky if the library overrides then become outdated and don't get updated so... shrug)

[true bronze]

Here's a wip bf emulator build with it. To rename a function you just add the existing name as an alias and then change the name as you usually would (the name is the one that'll always come out when decompiling and thus should be the most correct one)
So for example this function

    "Index": "0x300e",
    "ReturnType": "int",
    "Name": "EVT_FUNCTION_000E",
    "Description": "Address: 0x08A5FB18",
    "Parameters": []
  },

Would turn into (assuming for some reason A_NEW_FUNCTION_NAME is a good name)

    "Index": "0x300e",
    "ReturnType": "int",
    "Name": "EVT_FUNCTION_000E",
    "Description": "Address: 0x08A5FB18",
    "Parameters": [],
    "Aliases": [
      "A_NEW_FUNCTION_NAME"
    ]
  },

If something's already got an alias, you'd just add the current one to the list so all past names are always kept track of

That one's got aliases for all of the named P3P and P4G functions so in case there was someone using some ancient function names it should work

When I actually make a pr I'll probably do that for all the libraries (except P5 and P5R since they're special and have real names )

[dusk crow]

was able to give it a quick test and it seems to be working as expected 😎

tried renaming FUNCTION_0030 through FUNCTION_0033 and using the renamed names in one mod, and tried using the original names in a different mod; works exactly as I expected it to, no issues

this is also making me realize one of my names here is just wrong ("GET_MOON_PHASE"; it def isn't getting the moon phase lol)

[mortal yarrow]

wait, function_0033 is moon phase?

not time of day?

[dusk crow]

no, it's time of day i think

[mortal yarrow]

oh

[dusk crow]

i had it wrong originally is what i mean

[mortal yarrow]

oh lol

myb

[dusk crow]

no worries

[mortal yarrow]

you literally put it right there

sorry

[dusk crow]

ahhh dont worry bout it, it happens

[mortal yarrow]

it's been a long day 😭

[dusk crow]

yeah no kidding, im about ready to head to bed myself lol

like id test things a little bit more but. it's almost 4am i gotta get some sleep 🫠

[mortal yarrow]

same it's like 3:50 am for me

[inner sky]

wait

[simple timber]

[dusk crow]

(on the note of testing things im now wondering if aliases would also work for enums or not; like is it possible to change the "Agi" member in the BattleSkills enum to be a different name (IE idk Fire lol) , providing "Agi" as an alias so mods that use the original/older enums would still be compatible the same way as functions?)

(id test that but considering that swine didn't mention it as a thing + im going to bed here in like one minute im not gonna bother testing it since it probably wouldn't be worth the effort)

[true bronze]

It's not a thing rn but it could probably done without much trouble. I might look into it tomorrow

[dark tree]

yeah renaming functions to match P5 (official naming scheme) would help keep things consistent, and it would also facilitate porting scripts between games (like the time i ported shadow yosuke AI)

[simple timber]

with msg.tbl bmd merging, it seems like having two mods that edit the msg.tbl bmd cause r2 to break rather than having one overwrite the other (for p3p idk about p4g)

bungeecums

Jump

[Go to message!](#1173862131041566720 message)

#need help running mods on p3p this error pops up but idk what it means

[true bronze]

Oh, fun

Which two mods was it with?

[scenic sand]

ah, right, i need to add support for that file

i say like i didnt remember but i did, i just put it on hold bc the emulator in general isnt working

[empty galleon]

oh okay haha

[empty galleon]

localization fixes and useful descs

but i did some testing last night and it just happens in general with any msgtbl.bmd

[scarlet bolt]

Oh right I had that happen to me

It just crashes

So I turned off the former

[true bronze]

I love when people have problems and just don't report them 🙃

[empty galleon]

I only learned about this yesterday night 😭

[scarlet bolt]

I didn't recall to mention here haha...

[scenic sand]

i dont even have p3p dont look at me

[scarlet bolt]

honestly I should have recalled cause the mod page says they're better used in tandem

[true bronze]

Well, anyway, I'll try and fix it

Oh, actually I remember that I was having a crash with useful descriptions as well lol. I ended up just turning it off, never realised it had something to do with msg.tbl

[proper perch]

yeah that oddly started working again by changing nothing lol

[empty galleon]

thanks. i put out a quick update to localization fixes to just remove the msgtbl file until this is resolved bc idk how many people are using localization fixes and not useful descriptions

[lethal orbit]

https://github.com/Secre-C/PersonaEditor/releases updated PersonaEditor to properly read spr sprite/texture entries using the offsets in the file, fixing an issue where it sometimes doesn't properly read spr files created by the spd emulator

[scenic sand]

ive been meaning to ask but kept forgetting due to being busy

1. my unit tests have been failing, the actual has more bytes than expected and different bytes from expected, even though the messages seem fine. out of curiosity, i decided to do an ingame test with a modified ver of cbt, dumped the datDressHelp.bmd made by the emulator and compared it to the release ver of the mod and the only difference seems to be a missing tag at the start of the modded messages. the messages work correctly ingame btw. (not sure why script compiler gui doesnt label the start tag as just [s] on decompile but thats beside the point)
2. its probably me missing something really obvious but loose bmds seem to just be...ignored when i test ingame? the route appears in the log and then they never get mentioned again, and the files stay vanilla. paked bmds work fine though

actually now that i say it out loud that first one is probably just me, ill redo my msg files and see if that does it

[true bronze]

The extra tag is most likely just because you have a different library. It doesn't matter

[scenic sand]

hm.

[empty galleon]

found a bit of an odd issue that i've narrowed down to running through reloaded-ii with persona essentials and nothing else enabled - with certain skills in P3P PC (i only tested mabufudyne), there's this odd issue with certain file names where the windows error noise will play upon using said skills. I'm guessing it's because the files can't be read correctly; checking my game dump of p3p, the file names are fucked

[dusk crow]

ah yep, classic shift-jis moment no clue how one would address this issue sadly tho

if i had to take a wild guess i'd assume persona essentials or maybe crifsv2 does something with an incorrectly/mistmatched encoded file name which triggers the windows error sound as a result (but also doesn't cause any actual issues in the game itself?)

[empty galleon]

no actual issues in the game itself, yeah. it's just annoying to hear the windows error sound whenever I want to use bufudyne lol

I could try disabling Persona Essentials altogether to see if it's crifsv2 or not, let me do that rq

although i guess since persona essentials is what is used to actually replace CPK contents, I'm assuming it's something w that specifically

yeah it's persona essentials lol, silly thing to test in retrospect

[devout crystal]

you could fix that by renaming the files in the bed

though idk why youd edit bufudyne

[minor barn]

Ig it goes like "lemme try seeing if this file is being replaced/emulated", it can't read the filename, it screams about it (windows error sound), then lets the game load the original file no problem

[empty galleon]

literally just to fix the issue hahaha

bc it is really annoying!

[true bronze]

I'll have a look in a bit and see if I can fix it, hopefully it's not too hard to do

[empty galleon]

appreciate it :)
I'm about to become one of 3 people who are using P3P PC for a full playthrough so I'll see if I can catch any other issues that have been overlooked bc P3P PC is mid :')

would a list of textures with fucked names be helpful at all? I have them all pulled up here

[true bronze]

I think I'll be fine with just 1, if I need more I'll ask

[empty galleon]

okay

[true bronze]

Could you send the actual name of the file like you've got in that screenshot? I'm not sure how to get it, I just have the garbled stuff

[dusk crow]

ブフダイン氷白64.dds

is this what you mean?

[true bronze]

yeah, ty

The problem is it's already garbled by the time it gets to CriFsBinderFind since it's seemingly being converted to utf-8. I'm not sure where that's happening though...

[true bronze]

Ok, I've worked out where stuff is breaking down. When BindFiles is called the string's being marshalled as an LPStr which is ANSI so once it's in the hook for that the it's garbled and there's nothing that can really be done.
I'm not really sure what to do with this knowledge though, I'm assuming cri uses ANSI strings since they're marshalled to it a bunch of times in the code. I'd guess there's some way to make it not use them but idk how or whether that'd break other stuff...

(I'm not actually sure where the windows bing sound is coming from but I'm hoping if the correct characters for the file name are being used it'll somehow fix itself)

[true bronze]

Well, I've fixed one related problem. CriFsV2Lib had the encoding around the wrong way which is why when you extracted the files you got the garbled stuff instead of the Japanese characters.
According to TGE's template 0 is ShiftJIS and 1 is UTF-8, that's not what crifs is doing lol

[mortal yarrow]

that should really help that one guy who was trying to work on cutin mods

i forget his name 😭

[solar stratus]

Dniwe?

[inner sky]

no

[solar stratus]

oh hi dniwe!!!!

[scenic sand]

yo

[empty galleon]

that was my guess bc i only ever got the funny windows error sound when the garbled text appeared in the console

should be pretty easy to test at least (just use bufudyne)

[mortal yarrow]

@tawny sandal

progress being made that should help with your issue

[empty galleon]

I'm kinda surprised this didn't come up sooner ngl

was half expecting this to be some P3P port moment 😭

[true bronze]

The problem with them being extracted incorrectly did come up a while ago, it's just that no one bothered looking into it ig

[empty galleon]

ah, i probably missed it or something. well, thanks for looking into it :)

[true bronze]

Back to the loose bmds not working thing. If you can send me the code (whether on github or just the actual files) or something I can have a look

[scenic sand]

ill send in a bit, ill just finish eating first

[scenic sand]

https://github.com/oceanstuck/p5rpc.modloader
https://github.com/oceanstuck/FileEmulationFramework

it will be once it works

[true bronze]

cool, I'll have a look

What's an easy to test file I could use?

[scenic sand]

like for ingame merging?

[true bronze]

yeah

You said that loose files weren't being picked up, I was going to try that

[scenic sand]

pay no mind to the test messages being shitposts

theyre for faith rank 10, either ver

[true bronze]

bruh, those certainly are messages 💀

[scenic sand]

the test mod of all time

[true bronze]

So, it does work with bmds in paks?

[scenic sand]

yeah

[true bronze]

interesting, could you send a mod formatted for that

[scenic sand]

local copy of alternative divine judgment

[true bronze]

As I thought, the problem is you don't have a dummy bmd file. If you want to allow for not having a dummy file (i.e. giving the full path in FEmulator\BMD like pak and spd do) you'll have to change (or add) some code

[scenic sand]

oh

💀 i was about to say i didnt remember bf emu needing it but it did and i just forgot

[mortal yarrow]

just updated one of my upcoming ports for bmd merging 👀

glad to know it's being worked on and how it mostly works

all that's left is to recompile the dll with proper names and descriptions for configuration

[lethal orbit]

if you copy the spd section in p5ressentials you can make it so you don't need a dummy file unless it's a bmd inside a pak

[scenic sand]

Instructions are basically the same as replacing a msg in a bf, create a dummy bmd in P5REssentials/CPK or FEmulator/PAK then put your .msg file in FEmulator/BMD. Include only added or changed messages, and keep message labels for changed messages the same.

Note: Not yet support for those embedded bmds in tbls, and this build doesn't have SPD merging since I thought I should wait for that to leave code review first.

[arctic mauve]

So like I'm probably dumb and have no idea what I'm doing but the modloader inside of that tells me to go f myself lmao

I'm not looking for a solution or anything, I kept a back up of the original mod loader I just wanted to point out that this happens

[true bronze]

Is this with a mod you're making for bmd merging or just something old breaking?

[arctic mauve]

I think it's just breaking as I didn't even have a bmd mod on when I tried to run the game with this

[true bronze]

It's most likely got nothing to do with bmd emulator. Bf emulator was updated to fix a bug where it was using thee p5 encoding insteead of p5r recently, it's caused issues for a bunch of mods that were using Japanese characters where they shouldn't have

[scenic sand]

i could feel in my bones it was something small and dumb like that
programmer moment

[empty galleon]

any updates on that one issue with file names containing Japanese characters being read incorrectly?

[dusk crow]

if you check the githubs for CriFsV2Lib and CriFsV2Hook they actually DID get updated so that the issue would be fixed... unfortunately the github action failed to build a new release for CriFsV2Hook so until someone does it manually/the next CriFsV2Hook patch gets made the bug still exists with reloaded 🙃

[empty galleon]

oop lol, well it's good to hear it's been fixed regardless :)

[empty galleon]

manually built crifsv2hook to test it and it seems like the issue is still present at this time. i hope i didnt do anything wrong with that lol but if i didn't, seems like that fix didnt work for p3p at least

[dusk crow]

it still prints as garbled characters you mean? damn

i know the fix worked for crifsv2lib at least since the newest version of the crifsv2lib gui does read the names correctly now

[empty galleon]

I only tested in p3p itself (just used bufudyne) and looks like it's still messed up the same way
I cloned the github onto my pc and built from there so if that's not the right way to do it, oops 😭

[dusk crow]

that's very bizarre

from what i can tell looking at the v2hook repo it should be using v2lib to read the files so fixing the issue in that SHOULD also fix the issue in v2hook

dunno why it would still be failing in the same way as before

[empty galleon]

it is very possible that it is user error on my part

lol

but if what i did doesn't sound wrong to you, then idk 🤷

[dusk crow]

i would assume what you did ought to work

[true bronze]

I'm pretty sure it won't be fixed yet, the problem now is the path name is being marshalled as an ansii string (so Japanese characters become "?") by crifshook so I'll have to change that. I may do it tonight maybe

[dusk crow]

ohhhh ok, i assumed the issue was just with crifsv2lib

[dusk crow]

@empty galleon not actually fixed in crifsv2hook yet apparently

[empty galleon]

oof

well at least half of the issue is fixed!!

[dusk crow]

yep, it's something at least 😅

kinda nutty that following the saga of this issue is how i actually start using crifsv2lib for CPK extracting and stuff instead of what I was using before because i assumed it didn't have a gui program oop

[empty galleon]

😭

[dusk crow]

like i was using cripakgui before and then trying crifsv2lib's gui kinda shocked me with how much faster it loaded the file list lol; pmuch instant compared to the couple seconds it takes with cripakgui

[empty galleon]

yeah it's crazy fast

I dumped my P4G32 files a bit back before I rly got into modding bc I figured, oh maybe I'll want these

much slower lol

[dusk crow]

oh yeah i bet; haven't extracted a cpk yet but considering how much faster loading the file names is i imagine dumping is blazing fast haha

[devout crystal]

p4g32 is a different type of archive

neptunia pak vs criware cpk

That was completely re-d by TGE to the point you could add in your own files, just for 64 to release and make it completely redundant

[dark tree]

tbf it still sucked ass

file limit and all

but that wasnt tge's fault

that was the fault of the game being 32bit

[scenic sand]

figure i should work on msgtbl.bmd but my p5r specialization is showing
@true bronze what should index be? 4 for both? is the rest ok, its mostly copypaste from bfmerger

[true bronze]

I'm not sure off the top of my head, I'll check

Yeah, you're right, 4 for both seems right

[true bronze]

Ok, I've gone back to this now and after bashing my head for a few hours I've worked things out.
As I mentioned before, the problem is the string's being marshalled to ANSI when it's actually Shift-JIS. By switching it to use Shift-JIS the problem's seemingly resolved (no more dings).
There is one part that I don't think I can fix (without more effort) though. It seems that the console just can't display non ANSI characters at all so the Japanese of the name gets turned to ?s in the log, I am certain that what I feed it is the correct characters though so it's just visual.

I've kinda done it in the laziest way though, I've just assumed all cpks are using shift-jis (they definitely don't) so I'll have to go back and fix that so it will work with any (not that I mod any games that don't use shift-jis)

[inner sky]

i think this is an issue with windows cmd in general

[true bronze]

Here's an updated version if you want to confirm that it's all good for you as well. If you could actually replace the files and see that they change in game that'd also be nice

[inner sky]

cyrillic characters also turn into ?s if you don't add workarounds 😔

[dark tree]

shrimply boot the game using japanese locale

[true bronze]

Yeah, I wouldn't be surprised if that's the case. Not worth looking into more imo

[dark tree]

[empty galleon]

thanks much. i'll let you know how it goes
if it's just visual I think that should be fine, there's enough info to find out what file it is lol

...this is a new one

[true bronze]

oh, woops

[empty galleon]

i think my mods loaded though lol

[true bronze]

Use this one instead

[empty galleon]

ok, the p3p issue i originally reported is fixed now

i tried a texture replacement for bufudyne but i'm not sure it actually went through 🤔

[distant sorrel]

When I tried it just now, it seems to be intercepting the file, but instead of putting in the file I intend to replace it with, it loads a lower-res backup image.
Left- Unmodded
Right- Modded

Command line behavior seems as expected, though.

[empty galleon]

I'm guessing it's not loading tex2 but rather the original vita texture there

[distant sorrel]

Did your attempt load anything differently than usual?

[empty galleon]

p3p's broken textures as far as i can tell were for models only so it would have been pretty hard to tell

either way, my loaded textures did not work at all, i tested with something else just to make sure i hadnt fucked up

[true bronze]

Could one of you send the files you're using to test this so I can try and fix it?

[empty galleon]

sure thing

that's for P3P, it changes Bufudyne and the face of Kartikeya to distinctly different colors

[true bronze]

cool, ty

[true bronze]

I'm not having any luck so far, I don't even really understand how files are replaced let alone why these ones aren't 😔

With the old version where it dinged was it also unable to actually replace files?

[simple timber]

apparently not, I never personally tested it but judging by this thread, people certainly had issues with replacing files

lobster

i've edited some textures but things seem to work less well in the tv. follow-up and syoukan cutins are working just fine, so i'm sure it's not a file structure issue, but the edited all-out attack portraits aren't showing up in-game.
i'm sure it's not the file names because i took them directly from aoa with guns (https://gamebanana.com/mods/download/9641)
2_陽介冬.dds & 2_陽介夏.dds
^ i've also tested adding the boko folder directly from that mod to mine and they didn't show up either.

Jump

[Go to message!](#1169759465373966426 message)

#all-out attack cutins not displaying.

Download : all out attacks but with guns or something [Persona 4 Go...

gun pew pew brrr ping pong gzzrrrt gazruttt boom pow... A Persona 4 Golden PC (32 Bit) (P4G PC 32) Mod in the Just for Fun category, submitted by SeeEff

[true bronze]

Alright, that's sort of what I expected but it's good to know for sure

[scenic sand]

debating whether its worth the effort removing the need for a dummy bmd for loose files
the trade is youll have to recreate the path in the bmd folder like you do with spds, which seems like about the same amount of effort for the mod creator. like maybe it could help with sorting if you had a large mod with a lot of bmd edits (or if two files had the same name for some goddamn reason but im not aware of any case of that), but it doesnt seem more convenient than just putting them in one place like you currently would and using the searchbar if you have trouble finding something. idk, does anyone actually care

[empty galleon]

it just means one less file to worry about. which would be nice but it's probably not a huge deal especially if it is a disproportionate amount of work just to get it working

[scenic sand]

i did manage to get it working (by just copypasting spd emu code) but it didnt feel like i gained much except for a crash when i forgot to refactor my local copy of cbt lmao, so i just undid it

ill leave it as is unless people start asking for it all of a sudden

In the meantime, msgtbl.bmd support (i hope).

[empty galleon]

crazy, we were just talking about that

I'll test it in a little bit and get back to you on it lol

what happens if someone edits MSG.TBL to change skill names?

[scenic sand]

ill be real i havent tested on my own machine, ive just kinda edited some bfemu code and prayed

[empty galleon]

hm lol well I'll test and let you know

I'm curious if it's similar to other cases where MSG.TBL will be overwritten by the BMD edits

if so, that is a problem just because you have to edit the TBL directly to change the skill names... it's not in the BMD

[scenic sand]

would have to leave p4g to install for x amount of time, and set up a test mod, and the testing environment, and tbh i dont own p3p, and this is where it becomes obvious that im a bit overspecialized in p5r

[empty galleon]

hahaha it's okay, I can test for you

P4G and P3P should be similar anyways I just happen to have two mods I can test for P3P right now

(right now = after dinner)

[scenic sand]

i do like them all its just p5 sucked up all the autism lmao

[empty galleon]

that was P4 for me 😭 I wanna mod P5 sometime but I feel like people have already done lots with it at this point looking at the existing mods out there

[true bronze]

My opinion on this (which is probably different from most) is that using dummy bfs is "better" (it's what you'd expect if you read up on the documentation of FE framework) but I do recognise it's mildly inconvenient for mod creators (at least until you've gotten used to it).
I think it'd be a nice compromise if it could work either way. I'm pretty sure it shouldn't be particularly hard to do (assuming no one's particularly against it and you actually want to)

[scenic sand]

personally i didnt find recreating and navigating paths more convenient than the current dummy bmd setup
i could try setting it up to use either method although id have to think about that and tbh im procrastinating on other things i should be doing

[true bronze]

Fair, leave it be if you want. It's something that can always be added later on

[empty galleon]

I think at best you can say it's slightly more convenient but I'm personally fine with doing it the current way

[dark tree]

having a dummy file better communicates what's happening to people looking at mod files

dummy file means a file with name but nothing in it

[true bronze]

I'd prefer if spd, spr, and pak emulators also worked that way (since they're already out though they'd obviously have to support both though). Whenever they have to be ported for r3 I'll hopefully make that happen

[lethal orbit]

No

[true bronze]

What's left to do with it?

[empty galleon]

(or if two files had the same name for some goddamn reason but im not aware of any case of that)
it seems I have found one lol

take a guess which game it is 🙄

[true bronze]

There are plenty in p3p and p4g. Every field's bf is duplicated about a billion times (i.e. one for each field with the same major plus 1 extra)

[scenic sand]

bruh

[empty galleon]

this is why persona 5 is the only good persona game

[scenic sand]

lmao

[empty galleon]

i'm sure it'll happen one day 😔

[scenic sand]

curious how this is handled though

[empty galleon]

apparently you use routing

huh, no embed?

makes it easier if those duplicate file names are the same which i would assume to be the case, but i wouldnt know

[scenic sand]

if two bfs with same name have different contents and you have dummy for both, does it just die on compile or do you have to replicate path in FEmulator/BF

[true bronze]

Firstly, the duplicate bfs are exact duplicates, I don't know of any cases where there are two with the same name but different contents. Assuming there was a case though, you'd just use routing.
For example, say you have two bfs named f007.bf that you're changing (and they need to be done separately):

• field\pack\fd007_002.arc\f007.bf could be routed as FEmulator\BF\fd007_002.arc\f007.flow
• field\pack\fd007_003.arc\f007.bf could be routed as FEmulator\BF\fd007_003.arc\f007.flow
  Alternatively you could go more more specific and route one as FEmulator\BF\pack\fd007_002.arc\f007.flow (or even the full path if you want).

Basically, if you look from the right of the full path you can choose to go back any number of folders to the left and that'll give a valid route. So, every possible route for a bf in field\pack\fd007_002.arc\f007.bf is:

• FEmulator\BF\f007.flow
• FEmulator\BF\fd007_002.arc\f007.flow
• FEmulator\BF\pack\fd007_002.arc\f007.flow
• FEmulator\BF\field\pack\fd007_002.arc\f007.flow
  (Note that FEmulator\BF isn't actually part of the route, I just think it's easier to see writing it that way since it's the folder structure you'd use)

Which one you'd choose would depend on how specific you need to be (as not to catch both files in the one route) as well as personal preference (I generally prefer the smallest route since less files to navigate when making the mod, but there's nothing wrong with using the full route if you prefer)

Sorry that's a lot of words but hopefully it explains the concept well

[scenic sand]

i c

[lethal orbit]

I made the changes sewer asked for, waiting on him now

[cedar shuttle]

Ping me next time, I was actually unaware you were done

Since you'd randomly come back to it over and over and make commit here and there

I just thought you were taking your time 😅

[lethal orbit]

my apologies, I should've been more clear about it.

I realize now that I told you I was done before I started writing tests, but not again after

[cedar shuttle]

dajiobu

[empty galleon]

just got around to testing this one, it still hangs when trying to merge msgtbl.bmd from multiple mods (to be exact, it hangs around when the second mod with a msgtbl.bmd is loaded)

[scenic sand]

oh

i dont rly have an guess as to whats happening which is probably my sign to actually...learn p4g

ill try and give a look over the tbl merging related stuff when i have time although currently im trying to knock out some of my end of semester uni stuff

[empty galleon]

no worries. I don't think the log had anything useful but I can send it your way if you'd like it anyways

[scenic sand]

sure

[empty galleon]

this was technically for P3P but I think both games should be more or less the same in this regard?

here you go. It's loading the msgtbl.bmd in Localization Fixes before Useful Descriptions that breaks right now

[scenic sand]

weirdly, bmd emu doesnt seem to be printing anything to log...

what happens if you only have one mod editing msg.tbl? it works properly?

[empty galleon]

yeah

[scenic sand]

probably not gonna test rn unless my brain really cant seem to do the work i need to do, but im gonna try and set up things so i can test with p4g later

qq: does anyone know a mod that edits p4g aicalc bfs, i want a point of comparison

[true bronze]

culprit ai fix should

https://gamebanana.com/mods/437855

[scenic sand]

...think i have a guess from digging through the tbl merging code and old messages from this thread, though not calling it yet

[scenic sand]

oh sure test build, work fine with one mod on someone elses machine but then stall until r2 gives up and the game launches without mods on mine

sigh the dump at least looks correct

curious. testing with two mods also produces a correct dump

imma try and attempt a hotfix anyway ig and ill see what happens

[mortal yarrow]

do we still need bmd test packages >.>

i got one, i clearly can't test myself

it's for P4G 64 bit

[true bronze]

So have you worked out what the problem is? I'm now encountering what I think may be the same error when hooking the embedded bf in aicalc

[scenic sand]

i noticed the functions for replacing the bmd in p4g/p3ptblpatcher were using File.ReadAllBytesAsync, while the bf functions just use ReadAllBytes bc doing it async was causing problems
tried switching to ReadAllBytes to see if thatd do it but uh, still cant get it to work

the bmd is dumping but the pak is not btw

dont know if that info is useful

[true bronze]

That's weird, any idea why the paks aren't?

[scenic sand]

im not at computer so i cant check/send log, but id guess its dying either at the repacking step or (more likely) when actually working on the tbl

[true bronze]

Fwiw I've sort of worked out the problem with the embedded bf files. It seems to have something to do with the file size, moddaman helped me substantially reduce the size of the bf and it worked fine. I still don't really understand why that worked though 😔

[true bronze]

I don't really feel like working on it rn, I may come back to it later if you don't work out the problem with bmds (since I'm assuming it's the same)

[scenic sand]

oh!
i got it to run, and both bmd emu and pak emu produced a correct-looking dump

want to do a lil bit of further testing, but not right this sec...

[true bronze]

nice, hopeefully it fixes the bf problem as well, it's been stumping me for a while

[scenic sand]

i had culprit ai fix on alongside the (admittedly very simple) test mods
granted i didnt check the bf emu output, but if im understanding your issue the modded game wouldnt even start properly?

[true bronze]

Yeah, it either crashed on boot or when loading a save (depending on the size of the bf)

[scenic sand]

i havent touched the bf related code at all tbh so im not sure whats up, maybe culprit fix is too small to cause the issue?

[scenic sand]

(i also turned log level up for everything except bf bmd and pak emu since i had a bunch of things on debug level, but itd be real weird if that of all things had an effect)

(also this isnt related but for some reason restore last surprise as ambush theme shows up in my p4g mods even though it is obviously not for p4g? not sure why, maybe ill investigate when i have nothing better to do)

[true bronze]

It must have just been setup incorrectly, if you go and find it in the manage mods tab you can changee which games it's enabled for

[scenic sand]

i did look, its only enabled for p5r

[dusk crow]

just downloaded it now and the modconfig only has the p5r.exe as a supportedappid so it SHOULDN'T be showing up for p5r

[true bronze]

ok, that is weeird

it's not universal? (I mean in its config, obviously it's not actually universal)

[dusk crow]

seems like it got set as universal yeah

yep unchecking that and saving makes it only appear in p5r's list as it should lol

[scenic sand]

wack

[true bronze]

Well someone should tell the mod creator

[scenic sand]

could you send the relevant mods? so far everything on mine seems like It Just Works™️

[true bronze]

Sure, you need the version of bf emulator in there because it's got updated libraries and stuff.

[scenic sand]

oh, p3p
...i guess i can see if the bf emu build breaks it

not now though its uhh 3am here

uh for now have my current build of persona essentials

[true bronze]

Sorry, I thought it was clear it was p3p. I doubt the bf emeulator build has anything to do with it (or at least it's broken in the old one as well)

[scenic sand]

oh

[true bronze]

So, did you get the thing with the bmds fixed in the end?

[scenic sand]

hasnt broken on my machine thus far, granted my test mods were very small

and i was testing with p4g, i dont own p3p

[true bronze]

No worries, I might try your persona essentials version tomorrow. Maybe it magically fixes the issue

[scenic sand]

curious, did you do it
my finals are over so i have time again

[true bronze]

Oh right, I forgot 😅
I'll check now

It's not crashing on boot anymore 👀

But now it's crashing whenever I try to use tactics (what I changed)...
I'm not sure if that's the bf's fault though or something else

[scenic sand]

oh...

[true bronze]

Ok, I'm pretty sure it's not actually fixed. I just got lucky with it not crashing 😔

[scenic sand]

you said it seems like a file size issue, right?
so theoretically if i can just...make a big enough bf then i could test on my machine to see if it happens with p4g too (and a big enough bmd to test if it happens with bmds)

[true bronze]

Yeah perhaps

[dark tree]

you can just add empty procedures at the end and dummy messages in a bmd

[true bronze]

What I think was going on is other files in the pak were being overridden or something like that so when they try to get loaded it gets nonsense and crashed

It's not loading the AICALC file that caused the crash, it was when one of the files just after it in the pak was loaded

[scenic sand]

like 20 of these is probably good right

[dark tree]

should be fine lol, just take an existing bf and paste those at the end

should guarantee making it bigger

[scenic sand]

lol i just copied the files from culprit fix, nuked the actual culprit fix code and just held ctrl+v for a few seconds

(since its specifically an embedded bf issue)

[scenic sand]

no crashes from stuffing a million dummy functions into p4g enemy.bf so far...

[true bronze]

Might be something specifically with P3P

[scenic sand]

ah yes, the specific game i dont have, that also happens to be a famously lazy port

alternatives are that i havent actually hit something that would trigger the crash, or i need more dummy functions lmao

[cedar shuttle]

Bit late to the party (a few weeks) but I'm gonna review the remaining code for Spd/Spr

[cedar shuttle]

Your tests are commented (disabled), is that intended? @lethal orbit

[lethal orbit]

yes. I spoke about it with you in dms, but it boiled down to me not being able to get the tests to build because of conflicting package versions with the other tests, and it not being easily solved

[cedar shuttle]

eh, I'll fix it then

If the actual tests fail though that's up to you :p

[lethal orbit]

I was able to get my tests working (and passing) on their own, just not with the other tests

[cedar shuttle]

dajiobu, it ok

[cedar shuttle]

Did small cleanup, including bringing all dependencies to latest Reloaded.Memory. Almost done, will finish up next morning and merge.

[lethal orbit]

https://github.com/Secre-C/FileEmulationFramework/blob/main/docs/emulators/spd.md

[cedar shuttle]

Should be good for merging, @true bronze can have a doublecheck just in case :P

I'm gonna look at the Essentials side now.

https://github.com/Sewer56/FileEmulationFramework/pull/10

[cedar shuttle]

Damn, this took a lot of time.

Anyway, @lethal orbit can you re-open PR in essentials?
Make PR with spd-merging-new fork from current repo.

I accidentally merged then had to force push to unmerge.

I made small changes, so re-open PR with that.

I gotta go for now, but on the essentials side, there's mostly just 1 small change to make :p

I'll leave it to you to test ingame though, since this is not covered by tests.

[lethal orbit]

how exactly should I re-open it? do I just make a merge commit with the spd-merging-new branch and re-open?

[cedar shuttle]

push spd-merging-new to your fork/origin

and open a PR

:D

[lethal orbit]

Yep

You always have to if the spd is in a pak. Also if you're not adjusting pos/scale you don't need an sprt at all

[cedar shuttle]

Should make Emulator/BASE.CPK/BGM.AWB/00000.ADX a real syntax tbh

For expressing these.

The only reason I didn't do this, is because there were uncertainties

[cedar shuttle]

For example you don't know if BGM.AWB is a file or a directory here, from the perspective of a hypothetical CPK emulator.

BGM.AWB could either be a folder to include inside an emulated CPK, or another emulator's input.

Only workaround I could think is some syntax like ~!BGM.AWB, that realistically won't match against a real file, that can be used to indicate 'don't include this lol'

[cedar shuttle]

For Linux that's .

(dot)

[cedar shuttle]

(btw check feedback in PR @lethal orbit , in case you missed it, it's just a smol thing)

[lethal orbit]

[cedar shuttle]

Someone say Rust? :3

https://tenor.com/view/ferris-rust-rustlang-crab-cute-gif-26396486

[cursive patio]

So what else is missing?

[tight vector]

BMD and SPR merging are in progress. I think this thread could possibly be closed once they release. I can't think of anything else major that would be missing at that point.

[lethal orbit]

ctd is the same format as ftd, just with a different extension

It's not on the list, but I know mudkip also wanted merging for bustup param files

[true bronze]

Those kind of files probably wouldn't be emulators fwiw, imo they'd be just a part of Persona Essentials like tbl files

[lethal orbit]

yea

same with the other file types on the list probably

[minor barn]

Uuuh, kinda, even the texts ones differ from one to another

[lethal orbit]

the text ones have a value changed in the header indicating they don't have section data. It's just part of the format

[dark tree]

@true bronze pinging you since it came up earlier, I was trying to explain to someone the forcing procedure index thing with BF hooking and i noticed its not listed in the reloaded wiki for the usage notes

actually it doesnt even mention you can do function hooking lol

https://sewer56.dev/p5rpc.modloader/usage/#example-loose-bf

[true bronze]

It's in the bf emulator docs

https://sewer56.dev/FileEmulationFramework/emulators/bf.html#forced-procedure-indices

It does say here but I guess I should be more explicit and say you need to read those docs

[dark tree]

ah

a link would be nice

and thanks

[true bronze]

The Persona Essentials docs is more for where you put the files, the other emulators do the same thing where the main docs are in femulator

That is a link, it's a hyperlink

[dark tree]

ah

[true bronze]

Clearly it's not obvious though...

[dark tree]

but you see, I am simple brained

links are always blue

/s

[mortal yarrow]

would ftd merging be possible with the new reloaded features?

i wanna work on some mods but they all require extending an ftd table, meaning they wouldn't be compatible with each other

would merging those be possible?

[lethal orbit]

No one's working on it yet

[dark tree]

aren't some of them different formats?

I know one of them is a text format

so you cant really do one fits all

[lethal orbit]

It's technically one format, with a subformat determined by a value in the header

[dark tree]

a

give sauce...

is it mapped in the template

[lethal orbit]

Yea, but it's not labeled as such (so I could be wrong lmao)

[dark tree]

[lethal orbit]

The ftd template can make the distinction though so it's probably correct

[lethal orbit]

oh nvm it is labeled

[lethal orbit]

depends on how you do it, but under the hood it's really just 2 formats. barring the text format, every ftd has a different entry struct layout, but that wouldn't really matter as far as merging goes, unless you want really detailed merging that can merge changes in the same entry.

[lethal orbit]

also fun fact, aside from ftd and ctd, there's also ttd (for tutorials) and mtd (for thieves den menus) mtd has a different magic signature, but it's still the exact same format

FTD0 is the magic for ftds, ctds, and ttds, and I dare you to guess what the magic for mtd is

wTD0

[inner sky]

what the fuck

[cedar shuttle]

(btw am I supposed to get the stuff merged @lethal orbit @true bronze )

Nobody told me anything since deadlocks were supposedly fixed

[true bronze]

I'm not sure what's happening, I helped SecreC. fix the problem and that's all I know

I think he said he was going to work on documentation

[cedar shuttle]

I remember usage.md being updated in a commit; but I'm not really sure if that was everything

[lethal orbit]

@cedar shuttle

[cedar shuttle]

Ah, alright

all g then

I wasn't sure if that was all the docs you wanted to write

[lethal orbit]

Am I missing anything?

[cedar shuttle]

I haven't checked, I'll see soon. It's probably fine though to ship

[scenic sand]

i forgot what i was waiting on...
uh. went and stuffed dummy functions in friend.bf as well as enemy.bf, and p4g is still not crashing, so yeah this could be a common p3p L, or maybe i need like a fucking 1000 of these

granted, ive only been futzing around in the early game and maybe it dies at like...an end boss, so if anyone wants to try on their machine...

...none of this really matters to bmd emu though unless it turns out msgtbl.bmd has the same issue. idk what i was twiddling my thumbs over

[true bronze]

I wouldn't worry about it, I'll work it out eventually probably. bmd emulator's much more useful

[scenic sand]

i kinda want to do a test build that isnt incompatible with spd emu, even though i dont think it matters when it comes to pring it
maybe also fix the unit test

[true bronze]

that's fine, when you make the pr stuff will get merged

The two shouldn't interact with eachother anyway

[cedar shuttle]

I will properly investigate this when PAK Emulator is eventually ported to Rust.

👀

[true bronze]

That was the hope but idk now, I probably should just work it out now given Auto Heal also has the problem...

Speaking of porting stuff to rust, any idea when we'd be able to actually start that?

[cedar shuttle]

Nothing technically stops us from doing that today.

Windows API hooks require hooking library which I'm working on, but that's just a small part of the equation.

90% of the project, i.e. the main lib and actual emulators don't really directly talk with that, so you can technically start anytime

I was considering throwing almost half my paycheck as a bounty for someone to get started 😅

[true bronze]

I see, cool. I probably won't work on it just yet but I'll keeep it in mind when I'm looking for something to do

Damn, hopefully you don't need to do that

[cedar shuttle]

It's not wrong to give something back in return sometimes :p

[true bronze]

True

I more meant the half your paycheck part

[cedar shuttle]

I was thinking of putting up a bounty of around 100 USD per emulator on average, give-take (depending on work needed to be done).

And double that for BF emu, if script-tools are ported to native.

That would add up to around 1000 USD, which is roughly ~37% my monthly salary (after tax)

[true bronze]

ah ok

it sounds more reasonable when you put it like that

[cedar shuttle]

It's not a huge amount, but should be a decent incentive to get started.

[true bronze]

if script-tools are ported to native
can't wait 🫠

[cedar shuttle]

That honestly sounds like a nightmare 😅

[true bronze]

Yeah, I dread that lol

[cedar shuttle]

I'd like to eventually port the entire backend to Rust for R3. Including all middleware (e.g. CriFsV2Hook) for performance and portability reasons.

To enable mod development on more niche platforms (e.g. Switch), even if I don't plan on actively maintaining those ports.

[true bronze]

How would that actually work with switch?

[cedar shuttle]

I don't do console stuff myself; so I don't remember the exact bootstrapping method to get own code running on Switch.

That said, a platform port would eventually boils down to the following:

• Figure out how to load the loader itself before game code starts executing. Completely platform specific, may be trivial, may be hard.

Then for mod support:

• Port reloaded-hooks-rs. Add 3/4 platform specific calls to libc , or native API. Less than 10 lines of code.

And ideally (optional but recommended, as it expands hooking capabilities).

• Port reloaded-memory-buffers. Add an implementation of memory page walking so you can do proximity based memory allocation. (i.e. allocate some memory within X bytes of Y address). [Around 200 lines of code]

As for mod management, I plan to do client-server architecture. So the launcher (manager) just talks to a separate server written in Rust. This server may be on same machine, or another within the same house.

Assuming the target (e.g. Switch) has sockets (TCP) available in their libc implementation (Switch does), it should hopefully run just fine. Outside of having to override some default file paths, I don't expect many issues there providing you can make homebrew with sockets.

While I don't plan to deliver these esoteric ports myself, I'm structuring the general architecture so that it's possible to make them, for someone who's sufficiently skilled/knowledgeable.

[true bronze]

That sounds really cool, hopefully someone does work on it at some point

[cedar shuttle]

Switch will most likely have a port of reloaded-hooks-rs at least. The author of arcropolis (Smash Ultimate Modding Framework) wants to replace their hooking library with my stuff once it's done.

Speaking of hooks library, time to rewrite the x86 part using zydis instead of iced for assembler/disassembler

iced is too heavy lol, entire rest of library is ~40kB, Iced was 380kB

(Zydis is 23kB for features I need for reference)

big mistake, now I pay price

[cedar shuttle]

I'd like a basic mod setup at least to have around 350kB of code at least, that's my target in the long run.

For a more complex setup with multi-emulators and archive parsing like Persona, around 600kB

And around 2MB of RAM used; just because I can probably reach that ballpark if I aggressively optimize.

[vital narwhal]

cant wait for spr/spd

[scenic sand]

wondering if theres an advantage to spinning script library off into its own mod...

[true bronze]

The benefit would be slightly smaller file size if you have both bf and bmd emulators (since they wouldn't both need to include it) and it'd be a bit easier to maintain since you'd only have to update that "mod" instead of updating both emulators. I don't know if it's really worth it though

[scenic sand]

yeah i thought it might be a bit much in effort if bf and bmd enu are the only big things using it
the idea just happened to cross my mind while i was staring at visual studio a while back, and reminded of it by costume framework using it although not for anything big (just outfit descriptions) which would make 3 mods including script library

[scenic sand]

was about to go "fuck it, just pr" and then remembered i should edit usage.md

...ill do it in the morning

[true bronze]

So bmd emulator is ready?

[scenic sand]

i cant think of anything im missing off the top of my head (though it is late at night), unless i want to implement the option to not use a dummy file after all, and no one has reported problems that i can think of, so i guess i just add/check over documentation and fix the unit test?

watch me remember something glaring in the morning lmao

[true bronze]

Cool

That'd mean all the merging is done

Well when it gets merged

[scenic sand]

if uhh
-loose bmds
-bmds in paks
-embedded bmds
all work, then thats a finished mod i think

i guess theres the chance that that last one chokes with too big a size like that embedded bf issue, but i cant replicate the bf issue so idk if id be able to spot if there was a matching bmd issue anyway

[distant sorrel]

Hey, I've tried to check back on this thread a few times for any updates on the Japanese character parsing errors from a while ago, and I was kind of afraid to bring it up for fear I missed something.
So, just to make sure; the last update we had was that somewhat inconclusive look that AnimatedSwine had at the issue, no? Has anyone else looked at this since?

[true bronze]

Oh, I forgot about that. afaik no one else has looked into it (I don't know who else would)
I think you're right about the current state; I fixed it not working in crifs lib but it still didn't work with crifs hook.

[distant sorrel]

Ah, well. Shame, I suppose.

[true bronze]

What's happening with spd emulator? It's seemed like it was ready a bunch of times but still nothing...

(Sorry if the question's annoying but it does feel like it's been an eternity and I've gotten a bit impatient. If life's getting in the way then disregard it, it'll get done when it's ready)

[lethal orbit]

[lethal orbit]

I figured sewer was busy so I didn't want to bother him about it

[true bronze]

I see

Well I'll bother him for you
@cedar shuttle Do you think you'll be able to merge the spd emulator stuff sometime soon?

[inner sky]

i love bothering people

[cedar shuttle]

Yeah sure

[true bronze]

Cool

[cedar shuttle]

I'm just a bit slow to do things during work week, I procrastinate a lot, haha

And often get carried away for research purposes

Today I was procrastinating by looking at Slint https://slint.dev/ a native UI framework

It's very tempting to write the entirety of R3 with Rust and Slint. This is very fast, native and lightweight. I like it so far.

Problem is manpower, I lack manpower.

Since I won't be able to leverage any of the UI work from work anymore at that point, I'd have a lot of catching up to do.

For instance, reimplementing the panel system that took 3 months to create, that allows you to split and manage multiple pages in the app at the same time.

That's really the main blocker. But at the same time, I aspire for R3 to be portable everywhere, Avalonia on consoles is probably too hard of a step to take, but something like Slint may work, even if without hardware acceleration initially.

I'm probably gonna throw an announcement to get feedback when am done with hooks and finalizing the R3 spec

R3 really is an 'impossible project' for 1 person already. But if there's anyone who can do it, it's me.

[solemn geyser]

what do you mean portable everywhere ? like it could work on android or as a homebrew for hacked consoles ?

[cedar shuttle]

That is exactly what I mean

Porting to those platforms should be feasible (for single dev)

[solemn geyser]

You are a very mad man

That is quite admirable and terrifying at the same time XD

[cedar shuttle]

I don't care how long it takes, I want to make the best thing in category to have ever been conceived

If I'm not done for another 2 years, so be it

I will do it better than anyone else

and advance the science of modding forward

[solemn geyser]

Damn... that's beautifully poetic !

https://tenor.com/view/crying-mood-hamster-sad-emotional-gif-20645758

[cedar shuttle]

hampter

[solemn geyser]

You have my... encouragements !

[cedar shuttle]

Alright, I merged it

[inner sky]

[cedar shuttle]

@true bronze Can you check something for me? I don't know of a mod off the top of my head that fits the criteria.

The RegisterBf and RegisterSpd functions technically leak a handle (handle). Because calling InvalidateFile doesn't close it.

public void RegisterBf(string sourcePath, string destinationPath)
{
    var handle = Native.CreateFileW(sourcePath, FileAccess.Read, FileShare.ReadWrite, IntPtr.Zero, FileMode.Open, FileAttributes.Normal, IntPtr.Zero);
    if (handle == new IntPtr(-1))
    {
        _logger.Error("[BfEmulatorApi] RegisterBf: Failed to open bf file with Win32 Error: {0}, Path {1}", Marshal.GetLastWin32Error(), sourcePath);
        return;
    }

    Native.SetFilePointerEx(handle, 0, IntPtr.Zero, 0);

    var fileStream = new FileStream(new SafeFileHandle(handle, false), FileAccess.Read);
    var stream = StreamUtils.CreateMemoryStream(fileStream.Length);
    fileStream.CopyTo(stream);

    var emulated = new EmulatedFile<Stream>(stream);
    _bfEmulator.RegisterFile(destinationPath, stream);
    _framework.RegisterVirtualFile(destinationPath, emulated, false);

    _logger.Info("[BfEmulatorApi] Registered bf {0} at {1}", sourcePath, destinationPath);
}

Is there a specific reason for this? I actually don't remember off the top of my head.

I remember @lethal orbit saying things got borked unless they leaked the handle.

If you do new SafeFileHandle(handle, true), to pass ownership of the handle to FileStream, and then UnregisterFile this; I think we won't leak handle anymore.

/// <summary>
/// Invalidates a BF file with a specified name.
/// </summary>
/// <param name="bfPath">Full path to the file.</param>
public void UnregisterFile(string bfPath)
{
    _pathToStream!.Remove(bfPath, out var stream);
    stream?.Dispose();
}

We don't ever call InvalidateFile anywhere but this is technically a bug in the case someone did

[true bronze]

Oh, ok. The problem for spd emulator was that the emulated file was based on the file stream so closing it meant the emulated file wouldn't work (it'd try to read from a closed stream). For bf emulator it looks like I just forgot to close the file stream since it's copied to a memory stream so it shouldn't matter

[cedar shuttle]

They should both be based on FileStream :v, since it's on disk.
Copying to memory isn't worthwhile, because then it'll get swapped out to disk eventually due to sitting unused.

[true bronze]

Oh, I think I see what you're saying with new SafeFileHandle(handle, true, yeah that makes sense

Oh, that's a good point, why did I do that...

Well I understand the problem, I'll try and fix it

[cedar shuttle]

It's a simple fix, I can do it right now, just need someone to confirm it's not borked

here lemme do it

[true bronze]

Ah ok, well then yeah I'm fairly sure what you're saying should work

[cedar shuttle]

(technically we shoould test invalidate too, since nothing nowhere calls it, but for now it's probably ok)

There's a lot of duplicate code between emus; in Rust at least, that'll probably get deduplicated.

Either through macros or callbacks. Rust is smart and the compiler will inline callbacks (i.e. delegates), unlike C#, oop

[cedar shuttle]

Anyway, I pushed to main, see if I didn't break the world :v

[true bronze]

Will do

[cedar shuttle]

main is also updated in essentials, and has Spd merged in
(where I needed to make same change)

[true bronze]

One thing that's wrong, the spd interface project in Persona Essentials isn't referencing the one in Submodules, it's referencing a local one

Things seem to be working fine in game though, I think you're good

[cedar shuttle]

huh you're right

I don't know how that happened

[cedar shuttle]

Should be live, took a bit to get rolled out, I needed to update CI config, and make minor adjustments.

Essentials currently building in CI, SPD emu is live

[true bronze]

Awesome

[keen thorn]

So after this..what else is left

[slender trail]

Dumb question but what would be like. An example of this

[true bronze]

BMD and then maybe some binary files like FTD

[lethal orbit]

I'm still trying to think how ftd would be implemented...

[true bronze]

Of the use for SPD emulator?
The first one I can think of is colourful skill icons and colourful button prompts for P3P. They both edit the same spr so they are incompatible. With them updated to support spr merging you could use both without any problem
There are probably a bunch of other examples

[lethal orbit]

Ideally it would be something like tbl where you'd include the entire ftd and the emulator would merge the differences.

[keen thorn]

Ah

[lethal orbit]

the only thing I'm not sure about is what if a mod relies on an entry being left unchanged?

I had the same hangup with a potential Bustup_param merging implementation

though I guess that's something that tblmerging doesn't resolve either...

and ig it's more of an issue for bustup_param than for ftd since bustup_param is almost always edited alongside other files, whereas ftds are usually edited on their own

yea exactly

[lethal orbit]

The problem would be if a mod requires an entry to be unchanged. Lower priority mods could potentially change said entry and the higher priority mods wouldn't have a way to change it back

[honest charm]

pixelguin mod updates!!!

[lethal orbit]

they're pretty much exactly the same, just switch out spdspr with sprt and dds with tmx

use PersonaEditor (my fork specifically) https://github.com/Secre-C/PersonaEditor to find the sprite id/index

wdym?

nope

yep

yea sorry, I'm not great at writing docs tbh . also I wasn't sure if I was supposed to guide the reader through the whole process of finding sprites and stuff or just outline how to specifically use the emulator

but yea that's one way to selectively edit sprites. Another way is to edit the original texture without moving stuff around, and use the notation to patch sprites without sprts

https://sewer56.dev/FileEmulationFramework/emulators/spd.html if you're using the link from the announcement this goes into more detail

right click on the spr and click view/edit

[dusk crow]

I think that would only be for P5 SPD cuz those use dds textures while p3/4 spr use tmx textures right?

[lethal orbit]

yea

btw I should probably clarify, .sprts are only necessary if you're using something like PersonaSpriteTools to cut the sprites out of the texture. Otherwise you just name the texture spr_(spriteindex).whatever

you could do either. If you do the entire sheet then you don't need an sprt, if you cut out the sprite you will. I'd recommend using PersonaSpriteTools to do it for you both to guarantee consistency in placement and because the texture file needs to conform to a specific size

68x68 would crash the game lol

if you're editing a bunch of the sprites in a file then I'd recommend editing the whole sheet, and naming the texture spr_1_5_10-15 or whatever

yea

[honest charm]

why did you break sewer56's program pixelguin

how dare you

[lethal orbit]

[honest charm]

one or more errors occurred

[lethal orbit]

oh uhhhh are arc files an archive 🫠

[honest charm]

this labyrinth of lifeeeeeeeeeee

[inner sky]

did you limit pak files to specific extensions

pak can be anything

one pak in p5r is .tbl lmfao

[lethal orbit]

yea....

[simple timber]

[lethal orbit]

old versions didn't lmao that's why it was working

[inner sky]

i think if you add arc it'll be everything?

[lethal orbit]

I was just about to ask...

[inner sky]

i dont know any other extensions

but in theory they can be anything

[lethal orbit]

lmk if this fixes it

can you send your files?

I think the emu dies when there are non spd emu sprs in other mods

also the parent directory should be SPD not SPR

seems to work otherwise

[dusk crow]

LGTM

[tight vector]

Let's Get This Money

[lethal orbit]

check the output spr i guess...

it seems to work fine without checking for arc btw

probably because it's nested though

[devout crystal]

Is it a bad time to mention it looks like custom configs in mods are broken

P5IT be deathing for me

[lethal orbit]

is it an spd merging thing

[devout crystal]

Im pretty sure

if I disable any of my options

crashy

if they're all enabled

worky

[lethal orbit]

does the game crash or does reloaded throw an exception

[devout crystal]

crash

[honest charm]

(not) having so much fun

[lethal orbit]

I should get to bed before I get swarmed

[honest charm]

the swarm is coming

[tight vector]

we forget to check, whether our shit actually works or not, but hey here we come 🗣️🎶

[livid jolt]

for all the people who have been adding all those merging formats to essentials thank you!!!

[devout crystal]

https://gamebanana.com/requests/57025 I put up a request for a couple of the merging stuff I've been wanting

Specifically for p5it, if these existed p5it and ferm could be entirely character specific, so people could disable characters and not see them at all if they don't want. Which lets them merge with mods like ferm.

[devout crystal]

oh right lmao its .pdd not .shd

this is the third time ive done this

both

well, hopefully

[true bronze]

If it's a consistent format then in theory it'd work with all

[devout crystal]

Well uh

For p5r's it kind of switchess endianness like halfway through iirc?

in one of them anyway

Bustup bin is supposedly like piss easy (according to secrec)

[true bronze]

fun 🙃

[devout crystal]

bustup bin is the one I really, really want

That'd fully make p5it character seperation work which would be awesome

and any other bustup mods

[dark tree]

nah

what they did was

they took P5's boss pdd file

and then they just slapped another pdd file inside it (with a smaller header)

[devout crystal]

thats so awesome

[dark tree]

otherwise its the same format as P5

although p5 is big endian and p5r's is little endian

[devout crystal]

i have no idea what youre thinking of but no its not fine

one file has everything in it

so without merging anything ever overwrites other stuff

man i named shit horribly wrong huh

[honest charm]

that looks fun

[true bronze]

If you enable dumping emulated files in spd emulator's config you can check them that way (assuming it actually has that option, I've not checked)

FEmulator-Dumps in the game's folder

Yeah, that's how the emulator works iirc. It's expected

This looks very messed up...
First is modded, second is vanilla

It's just these ones that are messed up in game?

Weird

I imagine it'd be pretty obvious if others were broken

It could be that my copy of Persona Editor is wrong, I'll grab the latest of SecreC.'s

[lethal orbit]

Not at my PC but I think mine does the same thing

[true bronze]

Oh, ok

[lethal orbit]

I'll look into it tomorrow night

[next sparrow]

for some reason, my custom skill mod just completely broke. It happened all of a sudden and it had no issues before, now it's always giving me error messages whenever I load it.

I feel this is due to the bin merging and I can't fix it for the life of me.

[scenic sand]

i dont think you know what the word bin means here

[tight vector]

If you need help narrowing down the cause, better to create a thread in modder-help for that first and come back when you're certain rather than potentially derailing the conversation here

[next sparrow]

didn't know that belonged in modder help, sry.

[lethal orbit]

nvm I was wrong

[honest charm]

is this a

moment

:3

[lethal orbit]

it doesn't though cause I get the same issue with the Yen symbol

[dusk crow]

time for new emoji to drop then: sorta works on my machine /j

[lethal orbit]

ilfittommbihtsiig ||it looks fine in the tool on my machine but I have the issue in game||

I think the game isn't reading it correctly

meaning I'm possibly not building the file correctly

[honest charm]

I CAN'T CLICK ON THIS CHANNEL ANYMORE THE FUCKING SUPER REACTIONS ARE BREAKING MY LAPTOP

no :3

[lethal orbit]

I wonder if there's some limit to how many textures an spr can have...

I tried using PersonaSpriteTools to create pixels modded spr, and it had the issue. I then built it again, but added a line in the program to remove some other texture before it built, and the Yen symbol was fine...

seems like the more textures I add the more things start to get fucked up

still not sure if its a game issue or emulator issue...

it does, but I make sure to update that

even the ones added at the end start to break too 🫠

[lethal orbit]

yea I don't know anymore tbh. I fixed up the PersonaSpriteTools patcher to input/output an spr to be 1 : 1 with vanilla, and it builds modded sprs correctly, but it still has this issue...

I wonder if there are any other sprs in p4g with more than 32 textures

[true bronze]

I kinda doubt it, maybe it is just an issue with the game...

[lethal orbit]

guess it's time to make the emulator straight up patch the pixels into the original dds /s

[true bronze]

If you can't work it out I might have a crack at it tomorrow. I'll try looking at the code that's loading the sprites to see what's happening

[lethal orbit]

I'll look more into it tomorrow ig, but yea that would be really helpful

[lethal orbit]

Tried to do that myself but remembered I'm not smart enough for that I feel like I can really only understand p5 stuff lmao

[true bronze]

Oh, I forgot about it. I'll try tonight

[scenic sand]

pffffbttttt

[cedar shuttle]

Merging :P

[scenic sand]

ill figure it out sooner or later i just felt like showing this lmao

[lethal orbit]

did you ever end up looking into it?

[true bronze]

Sorry, I forgot again

I might try tonight, maybe...

[true bronze]

I will actually look today, I've got nothing else to do other than work on utoc emulator and it can waiti an hour or two (hopefully it's not hard to find out what's wrong)

[lethal orbit]

[true bronze]

To save me a little bit of time could you send one of the mods that was breaking it?

[lethal orbit]

check the Yen in the pause menu to see if the icon is just a white box

[true bronze]

Thanks

[true bronze]

Did you make a template for sprs?

[true bronze]

Are the sprites in order in the list of SpritePointers in the file or is the spriteEntryOffset the id of it?

(Going off of the names in your template)

Actually no, it couldn't be entry offset those numbers are way too big

That must be like position in the file

Well I see something weird. Looking at the template sprite 0 (which I believe is the yen symbol) is at texture id 25, but the yen symbol is in texture 0...
Unless I'm just reeading this really wrong that is

Ok nvm, that's how it is in the vanilla one as well. I just don't understand the template 😔

I'm just going to wait until you can give me a better explanation on what the stuff in the template means @lethal orbit

[vital narwhal]

https://canary.discord.com/channels/746211612981198989/1199776326408745071

[honest charm]

lmao? why link that here

[vital narwhal]

it has to do with spr merging

[mortal yarrow]

didn’t the spr merging like

add a new texture

and then disable the original ones call

and then call the new one

[true bronze]

Basically, yeah

[mortal yarrow]

i remember trying the patching thing they made and it was something like that

not looking forward to the SPR merging for P3P PSP ports

[honest charm]

bonq: playstation’s handhelds’ strongest soldier

[lethal orbit]

the yen symbol is sprite id 1, not 0, but yea it's weird that sprite 0 points to texture index 25 in the template but that doesn't reflect in personaeditor? hopefully I didn't get the struct wrong

[true bronze]

Then what's sprite 0?

[lethal orbit]

oh wait I was looking at the wrong file, sprite id 0 does point to 25 in personaeditor.

[lethal orbit]

a dummy sprite i guess?

[true bronze]

Ohh

I wasn't looking at the spr id field in persona editor

[lethal orbit]

it's all 0'd out so I assume it's a dummy

[true bronze]

mb, I was just assuming the first thing is id 0

[lethal orbit]

no worries

but now I'm concerned cause I have no idea why that sprite is being messed with since I'm not using any mods that touch it

[true bronze]

Well the sprite entry offset for it is different but the actual data at that offset is the same

Why would the offset change if you don't actually edit that sprite though?

[lethal orbit]

if I add a new texture pointer then the offest for everything that comes after would change

[true bronze]

Oh right

[lethal orbit]

figured this out, I was looking at the file from data.cpk instead of data_e.cpk which is different 🫠

[true bronze]

So that's the whole problem?

Or do you mean you were comparing the modded one to the vanilla one from data.cpk?

[lethal orbit]

this

[true bronze]

Ah ok

[lethal orbit]

so probably not the problem

[true bronze]

So you know, I did start looking into this yesterday and made some progress but I'm still not sure what the problem is. I'll probably keep going today, hopefully I can figure it out soon

[lethal orbit]

does it look like it might be a game issue?

[true bronze]

I'm still not really sure, when I looked at the file in 010 it seemed right so maybe it's a problem with the game

[lethal orbit]

I don't think it's an issue with the file either... from my testing it just seems like things break when there are more than 32 textures.

happens in p3p too unsurpisingly. adding 32 additional textures to a single texture spr breaks the first texture in the list

the issue goes away if I remove one texture

[honest charm]

white box

[lethal orbit]

Probably, but that's something every mod would need to include, and would destroy compatibility

[wanton cradle]

unless i'm misunderstanding something, i think this is a bug:

var bindedBattleFile = Path.Join(this.bindDir, "CommonBattle.bf");
this.bfEmulator.TryCreateFromBf(this.eventBattleFile, "CommonBattle.bf", bindedBattleFile);
this.criFsApi.AddBind(bindedBattleFile, "BATTLE/EVENT/SCRIPT/0000.bf", this.modConfig.ModId);

bfemu fails to create the file unless the route has been added previously through a file in FEmulator/BF.

if i force a dependent mod to load first, then the above works fine

[cedar shuttle]

Looking at the source, this.bfEmulator.TryCreateFromBf creates a file that's recognised by the emulator but doesn't register it to the emu input itself.

[cedar shuttle]

So you may need to call AddFile first

[scenic sand]

bmd stuff test build thats not incompatible with spd stuff, dont think i needed to do this though

[wanton cradle]

thank you also had to fix some extensions/move bf creation until after mods are initialized but it's working great now

i don't usually use bf emu, but i thought function hooks were merged or are functions limited to a single hook? when the flow file in AddFile has a duplicate hook as one from a another mod, I get an error about duplicates: Duplicate procedure declaration: void BTL_ACTSTART_hook()

currently using AddFile with an empty flow so the route exists

[true bronze]

Multiple mods can't hook the same procedure

[wanton cradle]

ah

[true bronze]

It generally doesn't go well when you try to lol. That's why stuff like custom sub menu exist, multiple mods hooking the same procedure generally requires manual work

In theory it might be possible but I'm not sure how useful it'd actually be. Script compiler has a feature called soft hooks that I don't think I've ever seen used (idk if they even work) that add the hooked code after the original function instead of replacing it. I've never really tried though since regular hooks are good enough for me.
In the cases where I'd use it it'd have to be donee manually anyway since it's usually something like multiple mods editing a selection (custom sub menu)

[true bronze]

You could probably do something similar to how tbls are merged in the other games

Either that or, if we get a way to edit them in json form, just include your changes in json and have it compile that

[true bronze]

It's been a while, what's happening with bmd emulator @scenic sand? It sounded like it was finished last I remember, was there anything left with it?

I ask because I was reminded about it when someone asked me about editing a bmd in p5r

(I also realise I was meant to help with spd emulator being broken... I'll get to it eventually 😅)

[scenic sand]

honestly been meaning to pr, im just busy irl and forget how git works sometimes
sorry, ill get to it

[true bronze]

No worries, take as much time as you need

I just wanted to know whether there was anything I (or someone else) could help with

[mortal yarrow]

if you need help testing i got a lot of mods i can test with

[winter arrow]

@scenic sand Looking forward to seeing it get PR'd when you have time, thanks for your hard work 🙏

[mortal yarrow]

anyone aware of any weird crashing problems with tbl merging still?

getting a weird error i can't place with some pretty normal TBL files

[tight vector]

Screenshot of error? When does it appear?

[mortal yarrow]

happemns with both persona.tbl and msg.tbl weirdly

despite working sometimes like it shows here

[tight vector]

Do you get a pop out windows type error with that too or nah?

[mortal yarrow]

yeah one second

hoping this isn't just some timeout error or some shit

[empty galleon]

was P3P/P4G TBL merging ever completed? I was under the impression it wasn't

[tight vector]

🤔 I don't recognise that one personally

[mortal yarrow]

haven't had any issues with my other tbl merging mods

skill fusion spells uses it

and so does find a friend and gift cheat sheet

[scenic sand]

apart from that one embedded bmd yeah

[mortal yarrow]

hope it's not a dependency thing or some shit

i got no idea what's wrong with it

it'd be fucked up if it were like

too many edits or something

because i got like 11 other skills edited in here

cleared cache

no dice

[scenic sand]

@cedar shuttle bmd merge pr

[cedar shuttle]

Yup I saw

w<

[scenic sand]

im honestly kinda embarrassed how long that took given it was like...done for half of it and then the other half was me not understanding git

but its pr'd now which is what matters

[mortal yarrow]

i can finish signature skills 64 bit port

[mortal yarrow]

did the method change at all since the last test? does it still require a dummy file like bf emulation?

[scenic sand]

yeah

to that second thing

[mortal yarrow]

alright i gotta port all my vita original mods so i can finally kiss that game goodbye

[empty galleon]

end of an era

[true bronze]

I'll have a look over the bmd emulator pr later today but as long as it's been tested and works it's probably fine

I assume it's still heavily based off of bf emulator and that works fine afaik

[cedar shuttle]

The code looked fine to me.

I did read through all of it after work today

Though I can't build it, it targets some version of Script Compiler that isn't up on main repo

I left a comment on original PR anyhow

[scenic sand]

yeah its almost all copypaste lmao

[scenic sand]

ill try to figure out what the hell is up with that, once im not panicking my way through two midterms
next week is break at least

[cedar shuttle]

Don't sweat it :p

Study goes first, don't get lost in the mist

[scenic sand]

my local and remote match, so id guess its that second thing
not entirely sure what im supposed to do, should i just pr my script tools fork into swines or...?

[cedar shuttle]

Whichever one is used in existing project

[scenic sand]

@true bronze boop or something

[true bronze]

Sorry, I thought I replied to this already lol
Ideally we'd merge all of the bf emulator and bmd emulator stuff into the main repo and just point to that but tge never merged my pr, presumably because that version doesn't work with his CI which is something I should actually look into.

For now I'd say just point to yours. I'll need to update bf emulator (and I'll probably do bmd as well) to support p3r soon anyway. By that point hopefully we can get it all merged into the main repo at which point I'll change the submodule to point to that

Also ik I said I would look at your code weeks ago but I think I just won't at this point unless there's anything in particular you want me to. Sewer said it's good so I'd trust that

[scenic sand]

ah ok

noted

[scenic sand]

(to be clear, id already made the pr by the time that was sent)

[true bronze]

What's holding this back from being released rn? Did you need help changing the submodule to point to your fork or something

I realise now that I wrote that reply out but never actually askeed what help you need in it

[scenic sand]

oh i had a pr to merge my changes into your fork open, i was under the impression thatd do it
(meant to bump this thread asking if you saw but kept forgetting)

if it wont, though, then yeah i guess making it point to my fork

[true bronze]

Oh I never saw that

I'll merge it now

I guess you don't get notifications for PRS to forks

It's done now. I didn't look at your code at all, I'm just trusting it's alright lol

Sorry, I had no idea I was the one holding this back

[scenic sand]

well it and none of the small handful of people who tried it caught anything wrong so its prooooooobably fine, right...?

[mortal yarrow]

i can stress test it sometime soon

[true bronze]

Yeah she'll be right

[mortal yarrow]

i got like 11 P4G Vita exclusive mods to port to PC that all use BMD merging

[true bronze]

If stuff breaks you can fix it. Users are the best test team lol

[mortal yarrow]

oh shit i have to finish the signature skills 64 bit port now, fuck

my plan semi backfired

its 99% working i just have to update the dll config stuff with the bmd merging part

[true bronze]

So is everything ready for a release now? It sounded like me merging your script tools pr the other day was the last thing

[scenic sand]

if it all builds then yeah

@cedar shuttle did that do it

[cedar shuttle]

I'll find out soon :p

Re-open the PR in any case

it'll fail CI if it's borked

[scenic sand]

the script tools pr?

[cedar shuttle]

Oh, let Swine handle that one

:v

[true bronze]

I already merged that

The File Emulation Framework one

[cedar shuttle]

Yeah that needs reopened because I accidentally merged back when I reviewed it

[scenic sand]

oh yeah lol

[cedar shuttle]

By forgetting to change remote when checking out oceanstuck branch

[scenic sand]

honestly that was my other guess, i just wanted to make sure i wasnt stupid

https://github.com/Sewer56/FileEmulationFramework/pull/18

[cedar shuttle]

I re-did the changes that were needed last time (fixing build on case-sensiitive filesystems, removing dead files), since it seems the commits were lost.

Script Compiler was pointing at the BMD branch, rather than the BF branch where the changes got merged into. I presume that's an error, so I changed that.

Just doublecheck here that both BF Emulator and BMD Emulator work ingame if built from source, since they share that specific submodule dependency. [I updated script compiler to BF branch]

And then I can merge, since code didn't change from last time
(so I already reviewed it)

[scenic sand]

got some homework to quickly knock out so ill do that first

[scenic sand]

i believe my current bf and bmd emu are both built from source though i think thats with script tools pointing to my branch

[cedar shuttle]

I accidentally merged again

By forgetting to update upstream branch

while intending to commit to PR's branch, aaaaa

Anyway, doublecheck the review comment here https://github.com/Sewer56/p5rpc.modloader/pull/31

And fix it up if needed.

Outside of that, it's good

[cedar shuttle]

(oh and ping me) @scenic sand @true bronze

Whenever you want a release of modloader

I already released FEF extension.

[livid jolt]

is bmd merging that last merging needed for all file types?

[true bronze]

It's the last file type that would need an emulator. Some people have requested merging for binary files like like ftds and things but it'd make more sense for them to happen in Persona Essentials like tbls

[scenic sand]

i dont see why not

[true bronze]

So... are we going to get a release?

[true bronze]

@cedar shuttle I would like to request a release 😄
Doesn't sound like there's any reason not to release it

[cedar shuttle]

Oh alright :v

[true bronze]

Thanks

Was there any reason you asked back then instead of just going for it?

[cedar shuttle]

Now make the release yourself :3

[true bronze]

lol

[cedar shuttle]

(you can do it)

[true bronze]

fair enough

[cedar shuttle]

Just push tag

And update ModConfig

[true bronze]

Yeah

[cedar shuttle]

I thought you'd want to be awake in case something went wrong.

[true bronze]

Ah ok

[cedar shuttle]

and/or would have wanted to make an announcement

[true bronze]

Did I just wake you up?

[cedar shuttle]

Nah

[true bronze]

ok, phew

[cedar shuttle]

My sleep schedule is borked, haha

[true bronze]

I had a feeling I might've

[cedar shuttle]

God knows when I nap sometimes

Changes every week

[true bronze]

Looks like you already updated the version in modconfig, I almost bumped it up again lol

It's building now

Do you update the gb page as well?

[cedar shuttle]

I can do it, I usually forget to.

I should probably make a bot someday to ping me when I don't do it.

[scenic sand]

weird, i set that reply to ping

[true bronze]

Well it has been announced, nice work oceanstuck 🙂

[cedar shuttle]

hello world

[scenic sand]

:)

[keen thorn]

ITS OVER BOYS GOOD WORK

[cedar shuttle]

On a misc note, can someone review this section of WIP Specification?

https://reloaded-project.github.io/Reloaded-III/Mods/Essentials/File-Emulation-Framework/About/

(calling out to devs)

I think I got parts that were missing from original C# mod's wiki, including info on how to actually make an emulator.

But there's still probably some improvements to be made.

---

Also need a section on when you want to recursively inject files.

I.e. You want to emulate a file which is inside another emulated archive. Without uploading the original unmodified game file.

I know we do it with 0 byte files here, but I've never checked how that works under the hood.

[true bronze]

It's specific to Persona Essentials in this case. Persona Essentials uses crifs to get the original file and then gives that original file to the emulator

I can't imagine any other way to do it unless maybe you had a full cpk emulator

[cedar shuttle]

one day

[true bronze]

[cedar shuttle]

I was curious about the recursive case, do we do that?

Emulated file inside file inside CPK

[true bronze]

I'm not really sure what you mean... I can't picture it

Is that like an emulated file inside of an emulated pak since the pak is in a cpk?

[cedar shuttle]

Yeah.

So like you want to basically emulate

• subfile.bin
  inside
• file.bin
  extracted from
• archive.cpk

The issue is we don't have a 'base' file for 'subfile' here.

[true bronze]

Ah ok

[cedar shuttle]

I wonder if the spec should have a solution for this. If we don't already have a decent one.

[cedar shuttle]

And we obviously don't want to reupload subfile from the base game files.

[true bronze]

I think it is still basically what I said before. For bf emulator at least it'd get file.bin from cri fs and then use pak emulator to get subfile.bf from that and pass that to bf emulator (the pak emulator api has a function for getting a file from a pak).
I'm not certain if pak emulator does the same thing, there's a chance it does something slightly different if it's a pak in a pak.
Either way, it's not something that's really generic. Maybe you enforce that archive emulators have an interface for getting a file from an archive and then use that

[cedar shuttle]

I was thinking maybe emulators during building would send a route for each file that's included up to FileEmulationFramework

And other emulators can try resolving.

Should be fast if we use Dictionaries (HashMap)

[true bronze]

Yeah, I think that makes more sense

So basically it's like those files were in an FEmulator folder

Oh no, probably a bit different

[cedar shuttle]

It's pretty much identical mechanism to what we're already doing to match paths in Builder(s).

[true bronze]

There's something that you use to search directories for files when an emulator is doing route matching right? I guess you could make that also search through whatever the emulators say they have in archives

[cedar shuttle]

Except we're firing routes from other emulators as opposed to file open.

[cedar shuttle]

I could but there's an easier way. Reuse code that matches when we open regular files.

There's only one caveat here, some perf loss on file open, obviously.

[cedar shuttle]

Well, kinda sorta can, to accelerate searches anyway.

Actual match will still need to be a loop

https://reloaded-project.github.io/Reloaded-III/Mods/Essentials/File-Emulation-Framework/Emulator-Development/Implementing-Emulator/#creating-emulated-file-builder

[true bronze]

Oh, I think I get it. So the emulator would try to open it as if it was loose on the disk then that would be pulled from the archive from the other emulator

So the emulator would also be "emulating" the files in its archives

[cedar shuttle]

Yeah pretty much.

[true bronze]

cool

[cedar shuttle]

Say you have an AWB file inside an AWB file.

When you're building the top level AWB file you call

framework.GetFile(innerAwbPtr, innerAwbLength, route) in the Build function.

The framework would then call to all emulators, same way as when you open a file on disk. If no emulator accepts, it just returns the input back.

Eventually you get an emulated file (from another emulator) back, if there's any input into it that matches the route.

[true bronze]

Makes sense to me

[cedar shuttle]

And of course this can recursively spider.

Only caveat now is perf.
try_create_from_route is O(N), if we do that recursively, it may get expensive.

Say you have an AFS with 1000 files.
You then call this API for each of the 1000 files, and this then will match against 100 files in some emulator builder each time.

Now you've done 100000 comparisons, oof.

If our archive/emulated format can't have subfolders, i.e. AFS, then we can thankfully reduce that to basically O(1). Because Matches is an EndsWith operation, we can have a HashMap of each filename (end of route) to groups.

But if we have a nested format that has subfolders, ooooof.

https://reloaded-project.github.io/Reloaded-III/Mods/Essentials/File-Emulation-Framework/Emulator-Development/Routing/#handling-subfolders

I.e. when you need to use matches_with_subfolder

I guess that same optimisation with inner folders might work.

Actually nevermind, same trick, dict match with last delimiter.

[mortal yarrow]

bmd merging israel