#Custom .pck packing with PCKPacker

1 messages · Page 1 of 1 (latest)

hardy otter
#

it's not automatic, but you can trace all assets to their corresponding file in the .godot/imports folder

#

might also want to thing about using zips instead of pcks, since anyone can unzip them to check for malicious mods and they load just the same

dense cedar
#

I see that ZIPPacker does not have anything to do with the compression level

#

Does this mean I wont be able to create a .zip file that wont be using any compression?

#

Oh, nvm, "stable" docs is missing compression level, while "latest" seems to have it

#

Does this mean its comming with 4.5?

hardy otter
#

could be

#

compression hasn't bothered us yet

dense cedar
#

Decompression speed is generally unaffected by the chosen compression level.

#

Interesting... So this means there is no reason to bother with compression being set to none

#

Thanks again, @hardy otter , gonna start working on porting this code as an editor plugin for my game

#

My current architecture makes it so the "core game" is also loaded as an optional mod :D

hardy otter
#

that's already an editorplugin. but of course it follows our mod structure

dense cedar
#

I am planning on doing things my way, I dont use GDScript for anything, for example

hardy otter
#

ah i see

#

good luck gdsalute feel free to ping for modding questions

dense cedar
#

@hardy otter I have returned with more questions :D

  1. Is it possible to somehow exclude absolutely everything from the build besides one script and one scene that is marked as current (So only export the Bootloader, in my case)?

  2. How should I go about testing file overloading when testing in the editor? After the game was build and all the mods got packed into dedicated .zip's its pretty explainatory - if you will load them one by one then the virtual filesystem will take care of itself with this. But what about when I am testing this from the editor? How would I make something like this?

fierce pendant
dense cedar
hardy otter
# dense cedar <@242761264198189057> I have returned with more questions :D 1. Is it possible ...

Yeah in the export tab you can select which files to exclude or which to include. If you only select the bootloader as include that should work. There might be issues with adding autoloads though since there is not function to do that at runtime.

Since I think 4.4 there are no more issues loading packs in the editor to test. You just kinda have to remove all the unpacked files. Might be worth just running all exported together from an /export folder outside the editor

dense cedar
#

Interesting. I will consider investigating this further

dense cedar
#

In order for that to be feasible, you have to constantly build .zips, place them into dedicated mod folders, and the launch from this setup for the file overloading to work

#

What about constant itteration? Let's say I will add/change/move/delete one file from the project

#

That would either mean I have to look for said changes and only repack these files in the resulting .zips accordingly

#

Or I would have to repack the entire project from scratch every time I want to run it from the editor (worst possible case)

#

I have been investigating into editor callbacks, and, for example I could not find a "has this file been moved" callback

#

Unlike in Unity's AssetModificationProcessor

hardy otter
#

If a project gets to the export phase I usually do

.
├── export/
│   ├── html
│   └── windows
├── godot/
│   └── main.tscn
└── .git/
#

Then you can put your other packs somewhere in the export folder too and test from there

dense cedar
#

What I mean is a bit different

#

I have a setup with a single central project

#

res://game/file.tscn
res://moda/file.tscn
res://modb/file.tscn

#

And this central project stores the game itself and it's mods

#

This is necessary to have a native workflow of using assets from the dependants within your own mod

hardy otter
#

In the editor you can just run all of them as files, no need to pack anything I think

dense cedar
#

I am not sure I follow, they will be in different paths, since they will be in their own separate root folders (game, moda, etc)

#

So no overriding could occur from this

hardy otter
#

Ah but you want to override?

dense cedar
#

That's why I was asking maybe I will do some magic trick with constant upkeep of res://moda

#

So it will always have an up-to-date moda.zip with all it's files at res:// root (stripping out the moda part)

#

And when you will try to run the game, you will load said .zips in correct order

#

Which in turn gives you overloads as you would expect them

#

My main constraint is that I need everything within the same project

#

So that the workflow will be as seamless as possible when developing and using assets from currently developed mods dependencies

#

So res://modb can use stuff from res://moda and res://game

#

res://moda can use resources from res://game

#

Idk if there is any other possible way of overriding asset paths

#

Without resorting to constantly packing/up keeping .zips/.pck to check for override stuff functioning as expected

#

Does Godot not allow you to remap files when running from the "editor" filesystem?

#

So I could look for identical files between mods, tell Godot to load those files from a different path, for example

#

Entirely eliminating the reliance on .zips to have working file overriding

hardy otter
#

Are you remapping your pcks on export so that the moda folder is removed or something? Because normally they would just load side by side and not replace anything since they are in separate paths

dense cedar
#

Maybe I will swap so that res://moda/overrides/game/test.tscn will override res://game/test.tscn

#

Haven't decided on the final system since I am looking into the most basic implementation

dense cedar
hardy otter
#

Yes

dense cedar
#

I will look into that, thanks for the help!

hardy otter
#

For our mods we want to keep all files separate and avoid overriding on load, that’s why nothing is remapped. But if a mod wants to, for example, swap a vanilla texture, they can use take over path to replace the vanilla one

dense cedar
#

Is it possible to change resource path without first loading it? By hooking into the Resource loader and telling it directly what remaps should happen or something

#

Resource.take_over_path expects me to first load the initial asset, but what is the point if I will be overriding it and expecting it to be loaded from the different place instead anyway

hardy otter
#

It only expects you to load the new one

dense cedar
hardy otter
#

Here’s how we do it

#

Take over basically changes the resource cache without changing files. You just need to load the new resource and put that into the cache

dense cedar
#

You are talking about overloading that happens from .zip/.pck

#

I am talking about emulating this within editor filesystem in order to not build said .zips on each change within filesystem

#

So that you could rapidly check these overrides while developing in the editor

#

I understand how to accomplish this with .zip/.pck, this happens automatically by just changing filesystem entries from one pack to another when loading them in proper order/sequence

#

I need to emulate this behavior without constant .zip rebuilds

#

You are saying that this can be accomplished with Resource.take_over_path

#

While this seems to work, it implies that I will have to load the replacement asset just to tell it manually to override a different asset

#

If I have a lot of assets that are heavy scenes that try to overload something

#

I will have to load them all up, tell them to take over some path, and then unload them

#

All just so that when the time will come for the original asset to be loaded - it will come from the provided path instead

#

I was hoping to eliminate this unnecessary loading just to remap the file

#

By passing something like ResourceLoader.remap_file("res://game/test.tscn, "res://mod/test.tscn)

#

So that when "res://game/test.tscn" will try to load, it will load from the "res://mod/test.tscn" for example

#

So no need to unnecessary load any of the files just to pass this path remap

#

I will try opening a PR about this, since it seems that this is already implemented with .zip/.pck that happens with overlapping paths

#

But is simply not exposed to be used for direct remapping of "FileA" to "FileB"

hardy otter
#

Though I was looking for an issue requesting a way to place loaded pck content in different folders, but I can’t find it right now

dense cedar
dense cedar
#

So now at the runtime you will just rely on default path override behavior of .zip/.pck's

#

And while working in the editor you can use this exposed method and virtually remap entire folder's worth of files in order to emulate the runtime behaviour with same path overrides

#

Allowing you to immediately see how overrides will behave without the need to build .zip/.pck's

#

Thanks for the pointers @hardy otter

#

Hope it will get merged soon

hardy otter
#

You can say that this sort of cache manipulation is good for modular games, ones that add update patches like some mobile games do, to get smaller initial downloads and for most mod loader

dense cedar
#

It exposes an already existing functionality that bypasses the need to load the resource first

#

Since all that the Resource class does in the original method is calling this newly exposed method by passing this->path as the "original path" under the hood anyway

hardy otter
#

The discord link? #1380126344440053861 message

#

Might want to structure that. Or at least copy it over

dense cedar
#

I am just going to remove that and leave what I already have written above it

# 
I have decided to expose `Resource::set_resource_id_for_path` that is already used by `Resource` class under the hood. 

The main benifit of this instead of using `Resource::set_id_for_path` is that you dont actually need to load the file just to remap it to a different path in the virtual filesystem, making it possible to remap entire folders worth of assets really fast by skipping the need to unnecessary load potentially heavy resources since `set_resource_id_for_path` communicates with the resource cache directly.
hardy otter
#

I’m asking for why. Who uses it this way. Why is this a thing worth exposing that might add some more maintenance effort since it can’t just be renamed and moved since someone might use it

hardy otter
dense cedar
dense cedar
#

Nevermind, after further investigation this seems to be a major case of overreliance on Resouce pointers

#

That PR of mine is only relavant to the process of remapping dependencies of scenes, and it seems to work in a different way that I was expecting it to

dense cedar
#

When you use something like Resouce.take_over_path you are permanently leaving the takeover resource loaded in memory for it to be considered as an override

#

If I have bunch of mods overriding textures of each other - this will result in zero issues during .zip/.pck loading, since those will only load the final destination resource when it will actually be needed

#

But when you try to remap stuff with take_over_path you are permanently leaving those override resources for the entire duration of the lifetime of the game

#

This seems like a major oversight, since take_over_path was added like a bandaid solution over the existing resource cache