Generic Asset Pipeline Upgrade

[vernal dove]

Transcript so far:

generic assets are like, a known quantity
feels like we could just add a GenericAssetCopyDependencyBuilder to Gems/LmbrCentral/Code/Source/Builders/CopyDependencyBuilder
and it would load the file and take note of any asset fields

thge asset builder SDK literally has a OutputObject funciton
and that OutputObject function collects deps from the object its outputting
in fact atom already wrote this for the same reason (atom uses a lot of generic assets)
https://github.com/o3de/o3de/blob/ce390d43e323a4d5a80eb209493dc4ea3c1a08fe/Gems/Atom/RPI/Code/Source/RPI.Builders/Common/AnyAssetBuilder.cpp#L190

We should extract that into the generic system (the COpy Dependency Builder) and then add an ability to register types rather than hard code it.
that "any builder" is literally a Generic Asset Builder

it

• loads the given file into an Asset<T>
• checks if there's a "converter" registered (there does not have to be)
• runs the "converter" if registered, or just leaves it as is
• extracts dependencies from the Asset<T> using the standard AssetBuilderSDK::OutputObject function
• pushes back the job product which will copy it

so yeah, if we want to do this, that'd be the task for it

1. take the atom AnyAssetBuilder, move it into instead the CopyDependenciesBuilder in the LmbrCentral (or somewhere else, could be aztoolsframework/it don't matter, just somehting always active)
2. add some sort of function that registers types to it that would be called by system components booting up somehow. Probably just add a ebus call into its Register function that collects anyone's response. Right now it uses a static array of "azasset", "attimage", "azbuffer". We'd want it to do something like EBUS_EVENT_RESULT(aggregateresult, SomeBus::GetGenericAssetExtensionAndtypeId)
3. I dont think there is a 3.

might need ot tweak it slightly since it expects an <any> at the root
but it has the right idea - load the asset, parse its deps, output it
I don't think a std::any has to actually be involved tho
like if we make a proper converter
you'd have to register it by asset type id though, like not just "*.myasset" but also typeid of myasset (the part that goes inside Asset, like Asset<myasset>)
then it could do a objectstreamutils::LoadObjectFromFile(..., typeid)
wish there were hours in a day or me's to code
its doing as simialr trick in anybuilder, its checking if the given typeid derives from baseclass Converter which has a Convert() function and if so, calls it. If no Converter present, it just leaves it as is (direct copy vs process)
AZ_RTTI can enumerate derived/base
it should be using a AZRTTI or any cast tho
its using reinterpret_cast

[red hearth]

I'm working on the Generic Asset Builder now

so far I ahve the builder working but now the hard part is extracting and emitting deps

[vernal dove]

If there could be a way to add a "register your dependencies" method to the generic asset handler, that would not be a bad workflow.

[red hearth]

I'm not sure what you mean? My current appraoch is like

before:

m_postFxLayerCategoriesAsset = aznew AzFramework::GenericAssetHandler<EditorPostFxLayerCategoriesAsset>("PostFx Layer Categories", "Other", "postFxLayerCategories");

after:

m_postFxLayerCategoriesAsset = aznew AzFramework::GenericAssetHandler<EditorPostFxLayerCategoriesAsset>("PostFx Layer Categories", "Other", "postFxLayerCategories", /*AutoBuildAsset*/ true);

and everything is handled for you

that is, the asset is automatically built into the cache, (no need to make a copy rule) and also scanned for any Asset<T> fields, which it interprets as dependencies

but this is very fluid, I can change whatever, add callbacks, etc

by iuts very nature of being a generic asset, which is guarinteed to be an objectstream object, it is guarinteed to be iterable via reflection, which means it can walk its tree and find all Fields of type AsseT<T>

what it can't do is know that some random AZStd::string field is a filename tho

may as well stew on it for a day or two and think about what callbacks and overrides we want to make available?

bear in mind the Generic Asset Handler is a runtime class, so as little as possible should be added there to do with building assets. Just a bool so far.

[vernal dove]

Ahh then yeah, you have it covered. Happy to have everything handled for me. 😆

[red hearth]

basically, the true makes the GenericAssetHandlerBase return a true for an override thats default false

theen, the asset builder gets all classes derived from AssetData (using serializecontext)
checks if there's a handler for them (*using asset manager)
checks if the handler is a generic asset handler (using azrtti)
checks if that value is true
uses the existing "extension" getter to retrieve the extension and register it

the builder loads the assetr (it must be loadable since its a generic asset, which is an objectstream file)

uses serializecontext to walk the tree in search of Asset<T> to register as preload deps

I can think of a few places where we might need some possible overrides/additions

but like, if you're wandering off the golden path and doing something really strange with the asset you probably are going to want to do your own builder anyway

[vernal dove]

Totally agree. Good to cover the 95% regular usage. Once you're treading farther you're likely experienced enough to handle things yourself. Similar philosophy to the class wizard templates. When I have some time I'd like to cover the underlying generic asset handler stuff, but until then, might as well let people do it without knowing how it works.

[red hearth]

with this change tho it would mean that the classwizar dno longer has to mess with setreg for assets

[vernal dove]

That would be good! I can definitely fix that once in.

[red hearth]

do you happen to know if we have any existing generic asset types that would bvenefit from this?

I can convert inputbindings to use this system, and I may as well, but, it doesn't refer to other assets

I don't think there is one

I might have to make one JUST to test.

basically I"m done and now need to test it

essentially I'm going to need a test where I can have a generic asset that has a ref to some other asset and then have it go thru the pipeline and then check if the dep is in the DB somehow

[vernal dove]

Yeah, I don't know of any native chained asset dependencies.

My go to for a blank set would be to:

• Use the ClassWizard to make 2 dataassets in the same project, back to back.
• Give one some arbitrary variables, float, string.
• Give the other the AZ::Data::Asset<FirstAsset> whatever variable.
• Build, load editor.
• Asset Editor > [New] > First Asset, add string, change float. Save
• Asset Editor > [New] > Second Asset, drag in FirstAsset file, save.

Either that will do when entering Runtime, or it might work when re-launching the editor.
Worst case, you need to create a basic component that uses AZ::Data::Asset<SecondAsset> to prime it in the runtime. (Make a Basic Comp, add the inclusions, variable ref, save, build, run.)

If appropriate, just delete the generated setreg, it's standalone.

[red hearth]

Yeah, but I'm trying to think of automated tests...

but without writing 20,000 lines of test mocks to test 200 lines of code

[vernal dove]

Oh hrm... That's curious for sure.

[red hearth]

I looked at the other asset builder dep test and holy cow its thousands and thousands of lines of code to test like 50 lines

the thing people who write these gnarly complicated tests dont seem to realise is that test lines of code are lines of code and can themselves have bugs and need maintainnence

so its counter productive to write a giant test rig of 20,000 lines of code to test like, a couple functions

[vernal dove]

Is it something where you could include the two files preconfigured, then have the test like... run them through the asset processor to get it to emit the "This has a dependency we don't know about" warning?

[red hearth]

WEll theres the extra issue of, I dont actually want to change AP behavior in any way when its not under test

but yeah the simplest way would be to somehow just add a new type of gneric asset in the asset builder tests

and ensure that it actually outputs the deps

and the product

like we dont want any test code active if you just, e.g. compile AP in normal profile mode

but I"ll take a fresh look tomorrow

[red hearth]

ok, I have at least a good test running

1. starts ap
2. runs AP Builder.exe on a generic asset with deps in debug mode
3. examines the output xml

you can run AssetBuilder.exe with a command line like

AssetBuilder.exe -debug (full path to asset) -output (folder of your choosing)

and it will run CreateJobs / ProcessJobs on that asset to "build" it

but instead of actually putting it in the cache or messing with AP it will instead output the results into the output folder, as well as the response to the job request, containing the resulting asset info like deps

so its pretty easy to write a pytest that just

1. starts ap
2. runs the above command line
3. reads the resulting XML file that contains the deps

[red hearth]

works fine and is pretty quick so I'll add it

also, writing the test revealed a new bug in general, see #sig-content

however, it occurs to me right now I have it as a new bool where you register the asset

maybe I should set a flags field instead

so we can put other options in there.
also, I'm currently scanning for only Asset<T> fields for auto-deps but maybe I need to include AssetId fields too?

that would require a much deeper scan

we can also add future extensions here, like being able to call a Process plugin optionally to process the asset, or to call a Get Deps call back for custom dep getting

[red hearth]

fwiw, the current new API is like

using MyAssetHandler = GenericAssetHandler<MyAsset>;
s_myAssetHandler = new MyAssetHandler("My Asset Friendly Display Name", "My Asset Group (ie, Graphics, Audio ...)", "my_file_extension_without_dot");
s_myAssetHandler->SetAutoProcessToCache(true); // if you want the system to register, build and copy it to cache and extract deps for you.
s_myAssetHandler->Register();

you will need to do this in the system component that registers them (ie,the above code)

void MySystemComponent::GetProvidedServices(AZ::ComponentDescriptor::DependencyArrayType& provided) override
{
     provided.push_back(AzFramework::s_GenericAssetRegistrar); // Activate me before things that need these registrations.
 }

but thats about it

so in terms of the templates, the regset files could be removed if the extra call to s_myAssetHandler->SetAutoProcessToCache(true); is set

and you do automatically get deps handled, currently, only for asset<T> fields but hmm, I'm tempted to add assetid as automatic noload deps

is that something people use, tho

https://github.com/o3de/o3de/pull/19572

Draft

[red hearth]

https://github.com/o3de/o3de/pull/19572 is ready

now features

• Automatic conversion of generic assets to binary form in the cache for faster loading and stuff
• Automatic registration of any dependencies found in the asset, including Asset<T> and AssetId

[vernal dove]

Is this still the format of the API?

[red hearth]

yep, not sure if I need to change it

note that the only new lines are s_myAssetHandler->SetAutoProcessToCache(true); and provided.push_back(AzFramework::s_GenericAssetRegistrar); (you had to do the rest already)

[vernal dove]

Yes, just double checking. I'll make the changes once this is merged.

Unfortunately, I don't actually know the depths of what you're doing. I don't know how I can arguably code review.

[red hearth]

eh, maybe you spot somehting

or learn something 🙂

literally just a new type of asset builder

like we have lots already, so it follows hte same pattern as all the existing asset builders

the only tricky stuff is, all asset builders register a set of file extgension patterns (like ".tiff", ".jpg") etc that they are responsible for building

to know what extensions to register, it just uses serialize and RTTI
it asks for all types derived from AssetBase (the baseclass for all assets)
Then it asks the asset manager for the handler for each of those assets (ie, the thing that loads/saves it).
It checks if they are derived from GenericAssetHandlerBase (again using RTTI). So it knows what assets there are that are handled by generic asset handler base

and genericassethandlerbase has "Get extension", "Get typeid" ,etc and the new "Is auto compile"

so it can use that to register the extensions it cares about

then in its "do the actual compilation and return the dependencies of it" function (ProcessJob) it just uses the raw serialize context to load the asset and an existing dependency extraction function. Since it knows its typeid, it can load it into a void* pointer, which is what the "extract dependnecies" existing function takes.

As a bonus it calls SaveObjectToFile on the data, which writes it to the cache as a binary file instead of a plaintext xml (still works as an asset, just more compact and faster to load)

[red hearth]

once this is in I might follow it up with a change to any of our existing generic assets that don't have builders (are just bering copied for example) and just opt them in instead

none of them have dependencies

which is why we just copied

but less setreg files is a win too

[vernal dove]

Op, there is no SetAutoProcessToCache() actually declared in the GenericAssetHandler.

It's only in the example comments.

[red hearth]

let me check

sorry its

        void SetAutoBuildAssetToCache(bool autoBuild)
        {
            m_autoProcessAsset = autoBuild;
        }

part of GenericAssetHandler<T> as part of that change

I'll uipdate the help comment

its AutoBuildAssetToCache

not SetAutoProcessToCache

my bad

[vernal dove]

No problem. Also, in the instructions you mention putting the dependency in provided, shouldn't it be Required?

[red hearth]

no its the other way around

you see, we want the thing that builds the assets to sort AFTER the things that provide the handlers

so the asset builder itself must Require the dependency

[vernal dove]

Ohhh.

[red hearth]

and the thing that registers the generic asset handler must provide it

[vernal dove]

I see, so you're sorting "all" system componetns to before.

[red hearth]

that will cause the builder to sort after all the registrars

yeah, the proicess actually collects all system components from all gems first

then it sorts them by provided/required/etc

and then it inits them all

then it activates them all

[vernal dove]

Perfect. 👌 Testing now.

[red hearth]

so if you want to be "After" another component, you must "require" it

so becuase the builder wants to be after all the registrars have already registered their handlers, it requires it. Note that theres also 2 levels of requirement

One is a soft requirement and the other is a hard requirement

[vernal dove]

Makes sense. I had thought that registrar was the other way around. Needing the build after makes a lot of sense.

Never personally used the logic that way, only relating to required layering. Pretty cool effect it has in other directions.

[red hearth]

REquired services are absolutely required and will fail if nobody provides it

wheras dependent services ar eoptional and will continue anyway if nobody provides it but still offer sort order if someone does

this is why the asset builder declares it as a Dependent service

so you can have 0 registrations but if there is one, it will activate after those providing it, but its okay to have 0

but yeah, you can use it in either direciton
"sort me after anyone providing one of these" = GetDependentServices
"Sort me after anyone providing one of these, and its an error if nobody provides one" = GetRequiredServices

"I provide this service" (no effect on sort order on its own) = GetProvidedServices

obviously in the case of these we expect there to be more than 1 so we do not publish it as an Incompatible Service

[red hearth]

now we just gotta figure out if theres some way to simplify it furhter

I think makiong it so "required system components" is done thru attributes is one improvement

but some way to register the component descriptors in the same file automaticalylu woudl be great too

[red hearth]

one day I'd love it if a "new component" was entirely doable in one cpp file (header optional) and one adding it to the list of files for the project

and I think its conceivably possible

but will require linker hax

[vernal dove]

That would be ideal. The less steps the better.

I will be glad to get the Data Asset simplified, my internal ones too.

[red hearth]

it would halp my vision of having a class wizard be an eventual "Additive" thing

the idea being, new project starts empty

no components, no dlls, nothing

you then add things to it via a wizard

as you need

first "Game" component you add -> adds a game module, adds the component to the module
first "Editor" component you add -> updates game component (or asks you to make one), adds an "Editor" module if missing, etc

[vernal dove]

100% that's the best part about O3DE in the context of C++ driven competitors. Starting from a blank canvas and building up is FAR easier to digest and plan than to start bloated and teardown.

[red hearth]

like the good thing is also that its not inachievable

like, you can actually do that, already, using a human

you can in fact make a project with nothing

and hten later decide you need a game module and add one

and then later decide you need a component in ther and add one, etc

its just making it simple enough that it can be wizard driven

right now you'd have to at least understand how moduels and components work and the macros in cmake that specify which modules are loaded by which application

which, granted, is not a lot to learn

and is learnable

but would be nice not to have to, to start that way

like a script only project is literally the empty project
It contains no modules. Contains some assets of course but, no code modules

[vernal dove]

Pushing that down the onboarding path is definitely better. The closer we can get the C++ barrier to entry line to flat, the less pain people will have in adoption.

When you're ready to do bigger things, then it'll be right there for you to finesse.

[red hearth]

Generic Asset Builder is an example of a decent step to make it simpler but without remving control

you can always set it to false and make your own builder

I wanted to set it true by deafult but every type we have already has its own builder and I don't want to break things

I would also like to go back and make it so that generic json assets work with the generic asset gui and editor

currently it only supports object stream xml, or object stream json, which isn't as clear or nice to use as source assets

can still output object stream binary for fast loading

[red hearth]

https://github.com/o3de/o3de.org/pull/2770 is ready for review, but like, I wish I could add a section there that slike "use the class wizard" (but its not ready yet)

[vernal dove]

Ahaha, you and me both.
I want to do an intensive, but don't want to promote something that needs a hackey means to install.

[red hearth]

qt6 is soon I think

[vernal dove]

Hrmm. When I remove my assets from the registry setreg, they are no longer... findable?

• The variable entry is there.
• The asset can be made in the asset editor. And saved.
• The assets cannot be found by the variable browser.
• Assets that exist, and are set on a variable are considered unfound.

(Ignore the squiggly lines, my source code isn't updated to match the codebase, but I'm using a built engine.)

[red hearth]

its probably related to the editor dll needing to be loaded into the asset builder

like, a cmake ly_create_alias issue

that code, "set auto build asset to cache" has to actually run in the asset builder

which means that whatever module contains that code must have an alias in the cmake file that aliases it to builders

[vernal dove]

Even if the editor module is dependent on the module it's set in?

[red hearth]

I dont understand that quesiton

[red hearth]

basically inside your cmake file there is a list of
ly_create_alias(...)

and those aliases tell the engine which tools (editor, runtime, builders, etc) to load the dll in

make sure that there is a ly_create_alias line that mentions .Builders to make sure the dll actually loads inside asset builders

[vernal dove]

# For the Tools and Builders variants of ${gem_name} Gem, an alias to the ${gem_name}.Editor API target will be made
    ly_create_alias(NAME ${gem_name}.Tools.API NAMESPACE Gem TARGETS Gem::${gem_name}.Editor.API)
    ly_create_alias(NAME ${gem_name}.Builders.API NAMESPACE Gem TARGETS Gem::${gem_name}.Editor.API)

[red hearth]

that wont work because API is a static lib and you cant laod a static lib

the alias is correct tho

but there needs to be somehting that tells it to load the actual module

the builder startup is psuedocode like

for every active GemName:
   does alias exist (GemName.Builders) ? 
      Yes ---> find out which TARGETs it says to load  --> load those targets

the .API aliases are just for convenience for others. They are good, but they don't control loading of dlls

[vernal dove]

I am not sure then, how to edit my targets. I am only aware of INTERFACE and STATIC
I did static because many of my classes are for inheritance.

I'm sure the data assets don't need extension, but then I'd put them in Private, which would also be static.

[red hearth]

thats fine, but you must have a {MONOLITHIC_MODULE_TYPE} in there somewhere

that generates the actual gemname.dll

NAME ${gem_name} ${PAL_TRAIT_MONOLITHIC_DRIVEN_MODULE_TYPE}

[vernal dove]

Ahh yes:

ly_add_target(
    NAME ${gem_name} ${PAL_TRAIT_MONOLITHIC_DRIVEN_MODULE_TYPE}
    NAMESPACE Gem
    FILES_CMAKE
        gs_audio_shared_files.cmake
        ${pal_dir}/gs_audio_shared_files.cmake
    INCLUDE_DIRECTORIES
        PUBLIC
            Include
        PRIVATE
            Source
    BUILD_DEPENDENCIES
        PUBLIC
            Gem::${gem_name}.API
            Gem::LmbrCentral
        PRIVATE
            Gem::${gem_name}.Private.Object
)

[red hearth]

this wordy macro ${PAL_TRAIT_MONOLITHIC_DRIVEN_MODULE_TYPE} is

• STATIC when compiling monolithically (this becomes a staticlib)
• MODULE when compiling normally (*ie, this becomes a dll)

so NAME ${gem_name} ${PAL_TRAIT_MONOLITHIC_DRIVEN_MODULE_TYPE} is just NAME ${gem_name} MODULE

and MODULE means "make a dll file"

only dlls can be dynamically loaded at startup

and gems are runtime things, you can turn on and off gems without recompiling the engine becuase they are dlls, like, plugins

[vernal dove]

So .API is added to the Shared module dependencies.
Editor.API has API dependence.
Editor Shared has Editor.API.
And the Editor Module is also aliased to the Builder.

[red hearth]

so all the ly_add_target clauses tell the CMake build system what to make and how they relate
the ly_create_alias clauses tell the system what to actually load

these two lines in that file

 ly_create_alias(NAME ${gem_name}.Tools    NAMESPACE Gem TARGETS Gem::${gem_name}.Editor)
 ly_create_alias(NAME ${gem_name}.Builders NAMESPACE Gem TARGETS Gem::${gem_name}.Editor)

they are telling O3DE that "when you are loading a tool, like the editor, use this module: Gem::${gem_name}.Editor

and when you're loading a loading a program like asset builder, also use that same module

so that actually looks correct

it will cause the TARGET Gem::${gem_name}.Editor described elsewhere in the file via the ly_add_target command to load in AssetBuildert

so from that we can at least confirm that your dll is probably loading in builders

the next thing to check is if your code is actually active in builders

for it to be active in builders, it must run the code that registers the generic handler in builders

which it loioks like your AudioDataAssetsSystemComponent is responsible for doing

which would imply that AudioDataAssetsSystemComponent is not created and activated in builders

the only way a system componnet gets created and activated in buidlers is if it has that special serialize tag

[vernal dove]

AudioDataAssetsSystemComponent is included as a system component in both modules, runtime, and Editor.

[red hearth]

specificaly, make sure it has this

        serialize->Class<AudioDataAssetsSystemComponent , AZ::Component>()
            ->Version(0)
            ->Attribute(AZ::Edit::Attributes::SystemComponentTags, AZStd::vector<AZ::Crc32>({ AssetBuilderSDK::ComponentTags::AssetBuilder }));

in its reflect

[vernal dove]

Ahh, it definitely does not have that.

[red hearth]

this is a bit of hax but the problem we encoutnered is exactly this sitaution

people reusing their editor module or runtime module, in builders

builders are headless, no-ui, background multithreaded CLI apps

[vernal dove]

I will need to update the Class Wizard to factor this.

[red hearth]

peopel were including thier like render stuff

and activating them in buiilders

and their physics, and audio and activating them in builders

and builders were doing things like initing the sound card

so we had to make it so that system components ignore the "Get required system components" list in builders

and are opt in only

I doint like it but ... the only other way would be to require people to make a seperate module (dll) for builders

[vernal dove]

Yeah, I'd only want to do that if it came standard with the Default Gem template.

[red hearth]

yeah exactly

longer term I'd like to explore removing GetRequiredSystemComponents and using this kind of tagging for runtime

its a vector already so it'd be pretty cool to do like

    ->Attribute(AZ::Edit::Attributes::SystemComponentTags, 
               AZStd::vector<AZ::Crc32>(
               { AssetBuilderSDK::ComponentTags::AssetBuilder,
                 AZ::ComponentTags::Runtime,
                 AZ::ComponentTags::Tools,
                 AZ::ComponentTags::Servers, 
              ...
}));

but for now assetbuilder is just special

because 99.9% of the time you dont want your system component actually initializing, grabbing GPU resources, initializing the audio, starting to tick physics, preparing editor gizmos, etc, in the NUM_CPU_CORES * AssetBuilder.exe it laucnhes

so assetbuilder.exe ignores GetRequiredSystemComponents and ONLY creates system components tagged with that attribute

make sure you also

void AudioDataAssetsSystemComponent::GetProvidedServices(AZ::ComponentDescriptor::DependencyArrayType& provided) override
{
     provided.push_back(AzFramework::s_GenericAssetRegistrar); // Activate me before things that need these registrations.
 }

[vernal dove]

Yes, do have that last bit already. 👌
Going to test right away here.

[vernal dove]

So I'm not sure if the AssetBuilderSDK is Editor only, in the sense that including a dependency to it would cause runtime issues.

But I set up my AudioDataAssetSystemComponent as such, using a manual entry for the tag name, and now the assets aren't even being brought up in the asset builder at all.
It used to just be a successful job, now there are no jobs whatsoever.

[red hearth]

that all looks correct, tbh

not sure what the issue is, this is why I wrote autotests

the autotests use the same pattern and prove that it can work - if it didn't work, then the test would fail, since the test ensures that not only does the job run, but it populates the dependencies correctly.,

the only thing I can think of is that its somehow still not loading it in builders

and we can check that

the easiest way is to make sure your actual dll gets loaded in assetbuilder, is to start the editor (without debugging it in vs, just "start without debugging) and then once its launched do "Attach to process" in the debug menu and attach to AssetBuilder.exe - any of them, they should all load all the gems

You can then look at the "modules" window in the debug menu to see all the dlls loaded by it, and see if your gem dlls is there

if we can verify that the module is loaded, the next step would be to debug an asset builder

to do that you'd run assetprocessor.exe yourself on the project ( you can launch the editor, or mnanually luanch it with the --project-path= command line option, just to have it running in the background

then youd set the "Startup project" as assetbuilder and edit the launch command line options to add
-debug (full path to one of the example assets you'd liek it to try to compile) -output (full path to some folder) --project-path=(path to project)

running that would let you put breakpoints in your builder executable to see your component activatre()

[vernal dove]

Using VSCode:

I attached to AssetBuilder.exe while Editor was open, searched GS_Audio in Debug Console. The only entry is:
AssetBuilder.exe (155244): Loaded 'D:\OffLocalDev\gs_play\build\windows\bin\profile\GS_Audio.Editor.dll'. Symbols loaded.

Launched AssetBuilder in debug while Editor is open.
Commands:

"--debug",
"D:/OffLocalDev/gs_play/Assets/Audio/EventsLibraries/TestAudioEventLibrary.aev",
"--output",
"D:/OffLocalDev/Builds/AssetBuilderOutput",
"--project-path",
"${workspaceFolder:gs_play}"

Output in Terminal:
Has trace warning for the MiniAudio SoundAsset.

C: [Trace] = C:/GS_Sys/GitLab_Builds/pJK2BkzC5/0/engines/o3de/Code/Framework/AzCore/AzCore/Asset/AssetManager.cpp(743): 'void __cdecl AZ::Data::AssetManager::RegisterCatalog(class AZ::Data::AssetCatalog *,const struct AZ::Uuid &)'

C: [Type] = Trace::Error

E: AssetDatabase: Asset type {7FEF8671-760F-4EBD-91E7-57AAF3EEF1CA} already has a catalog registered! New registration ignored!

AssetDatabase:
==================================================================
AssetDatabase: Trace::Error
 C:/GS_Sys/GitLab_Builds/pJK2BkzC5/0/engines/o3de/Code/Framework/AzCore/AzCore/Asset/AssetManager.cpp(743): 'void __cdecl AZ::Data::AssetManager::RegisterCatalog(class AZ::Data::AssetCatalog *,const struct AZ::Uuid &)'
AssetDatabase: Asset type {7FEF8671-760F-4EBD-91E7-57AAF3EEF1CA} already has a catalog registered! New registration ignored!
AssetDatabase: ==================================================================
C: [Trace] = C:/GS_Sys/GitLab_Builds/pJK2BkzC5/0/engines/o3de/Code/Framework/AzCore/AzCore/Asset/AssetManager.cpp(674): 'void __cdecl AZ::Data::AssetManager::RegisterHandler(class AZ::Data::AssetHandler *,const struct AZ::Uuid &)'

C: [Type] = Trace::Error

E: AssetDatabase: Asset type {7FEF8671-760F-4EBD-91E7-57AAF3EEF1CA} already has a handler registered! New registration ignored!

AssetDatabase:
==================================================================
AssetDatabase: Trace::Error
 C:/GS_Sys/GitLab_Builds/pJK2BkzC5/0/engines/o3de/Code/Framework/AzCore/AzCore/Asset/AssetManager.cpp(674): 'void __cdecl AZ::Data::AssetManager::RegisterHandler(class AZ::Data::AssetHandler *,const struct AZ::Uuid &)'
AssetDatabase: Asset type {7FEF8671-760F-4EBD-91E7-57AAF3EEF1CA} already has a handler registered! New registration ignored!
AssetDatabase: ==================================================================

Relevant Logs:

Info: Creating entity with scene system components.
SceneAPI: AssImp Import Context was registered.
AssetBuilderComponent: RunDebugTask - running debug task on file : D:/OffLocalDev/gs_play/Assets/Audio/EventsLibraries/TestAudioEventLibrary.aev
AssetBuilderComponent: RunDebugTask - CreateJobs: True
AssetBuilderComponent: RunDebugTask - ProcessJob: True
AssetBuilderComponent: LoadBuilders - Called LoadBuilders for [D:\GS_Sys\Engines\GS_Play_Engine\bin\Windows\profile\Default\Builders] - SKIPPING
Info: Skipping 'Scene Builder'.
Info: Skipping 'Shader Asset Builder'.
Info: Skipping 'Lua Worker Builder'.
Info: Skipping 'Material Builder'.
Info: Skipping 'Shader Variant Asset Builder'.
Info: Skipping 'Shader Variant List Builder'.
Info: Skipping 'Qt Translation File Builder'.
Info: Skipping 'Precompiled Shader Builder'.
Info: Skipping 'Prefab Builder'.
Info: Skipping 'MotionSetBuilderWorker'.
Info: Skipping 'Pass Asset Builder'.
Info: Skipping 'PhysX Material Asset Builder'.
Info: Skipping 'EmfxWorkspaceBuilderDescriptor'.
Info: Skipping 'AnimGraphBuilderWorker'.
Info: Skipping 'Script Canvas Builder'.
Info: Skipping 'UI Canvas Builder'.
Info: Skipping 'Generic Asset Builder'.
Info: Skipping 'Atom Image Builder'.
Info: Skipping 'Benchmark Asset Worker Builder'.
Info: Skipping 'XmlBuilderWorker'.
Info: Skipping 'SchemaBuilderWorker'.
< More skipping >

Then ends.

• It does not fire the AudioDataAssetSystemComponent breakpoint.
• It does not put any output in the designated folder.

[red hearth]

I guesss this is progress - I assume the GS_Audio.Editor.dll is hte one you expect to have that code in it, that contains AudioDataAssetSystemComponent

so the fact that it shows it loaded means it is laoding it and execing its module

in its module class, you could put a breakpoint in the module construcotr where it does the component descriptors

[vernal dove]

re: GS_Audio.Editor.dll
I assume it would be, although it's through the dependency to the non-editor targets, and to dependencies on the static targets.

[red hearth]

because like, yeah, its not going to be able to know about those generic asset handlers if they are not registered
and if the activate() on the system component is not happening, and that's what registers them, then that's the issue to chase down

[vernal dove]

For additional info:

I had to add to the system component, because I was getting an error from it trying to copy:
AZStd::vector<AZStd::unique_ptr<AZ::Data::AssetHandler>> m_assetHandlers; with it being unique pointers, going wrong.

AudioDataAssetsSystemComponent() = default;
~AudioDataAssetsSystemComponent() override = default;
AudioDataAssetsSystemComponent(const AudioDataAssetsSystemComponent&) = delete;
AudioDataAssetsSystemComponent& operator=(const AudioDataAssetsSystemComponent&) = delete;

I'll give the rest a look.

[red hearth]

the "debug" flag to asset builder makes it pretend to compile the asset so all the usual machinery will happen - activation, process job, etcv

Info: Skipping 'Generic Asset Builder'. means that GAB had no patterns registered for that file type

which means that the registration for that extension isn't present

which means the system component did not activate

but its hard to tell why

[vernal dove]

Okay, in order:

• Hit AudioDataAssetSysComp Reflect breakpoint
  • It fires 3 times.
• Whole build process fires. Outputting everything I already shared.
  • AudioDataAssetSysComp Reflect fired 3 times after.
• Finishes Execution.

I dont get the AudioEditorModule constructor breakpoint.
I don't get the GetRequiredSystemComponents breakpoint.
I don't get the AudioDataAssetSysComp Activate breakpoint.

[red hearth]

so the AudioDataAsssetSystemComponent reflect is being called

but that would only happen if someone, somewhere, returns its component descriptor in a module. Thats what causes reflect to happen.

the DLL is loaded
the entry point funciton is found
the entry point function is literally "return new (ModuleClass)"

[vernal dove]

The non editor module constructor fires once.

[red hearth]

regular module?

like, the one for the runtime?

[vernal dove]

Yes

[red hearth]

that shouldnt' be loading in builders, though, right?

I assume you're debugging the builder

[vernal dove]

But does not fire GetRequiredSystemComponents

The editor module by default inherits from the runtime model.

[red hearth]

hmm mayube thats the issue

let me think about this

you see, theres that AZ_DECLARE_MODULE_CLASS or whatever in the module cpp

[vernal dove]

So the runtime descriptors get fired, and then the editor descriptors (if there are any editor specific ones) fire next.

[red hearth]

let me think a moment, about what it means to inherit module class

the way the actual dynamic loading works, is that there's that macro
what that macro is doing is its declaring a DLL entry point extern "C" function that when called does return new moduleclass()

when o3de loads your gem dll, it searches the DLL for that known named entry point

and it invokes it, expecting it to return a pointer to a class derived from AZ::MOdule

which return new moduleclass() does, as long as moduleclass is derived from AZ::Module

then it loops over all the descriptors in the returned object's descriptor list, and invokes componentclass::reflect()

[vernal dove]

Maybe the stacking non-editor targets as dependencies in the cmakelist is causing redundant things? Not really sure about any of this boilerplate stuff.

The Editor module inheriting from runtime IS standard default gem template.

[red hearth]

that kinda looks fine to me actually

well the cpp does

looking at the cmake now

[vernal dove]

I brute force a lot, just to ensure things communicate.

[red hearth]

basically this line

AZ_DECLARE_MODULE_CLASS(AZ_JOIN(Gem_, O3DE_GEM_NAME, _Editor), GS_Audio::GS_AudioEditorModule)

essentially becomes

extern "C" { 
  dllexport CreateModuleClass() { return new  GS_Audio::GS_AudioEditorModule() };
}

when o3de loads this dll, it goes "Find the function called CreateModuleClass in the dll"

then it goes m_moduleClass = invoke that function

(it has a class for each dll loaded that has a m_moduleClass member)

this would trigger the constructor of the moduleclass since its calling new

sounds like theyh key here is AudioDataAssetsSystemComponent is not being created, or activated

inside builders

[vernal dove]

Could there possibly be an issue with the Install engine build output? I am running this project off of a prebuilt engine

[red hearth]

uh, shouldn't be, as long as its up to date.

it has the "generic asset builder" so its gotta be new enoguh to have that

GS_AudioModuleInterface::GS_AudioModuleInterface()
    {
        // Push results of [MyComponent]::CreateDescriptor() into m_descriptors here.
        // Add ALL components descriptors associated with this gem to m_descriptors.
        // This will associate the AzTypeInfo information for the components with the the SerializeContext, BehaviorContext and EditContext.
        // This happens through the [MyComponent]::Reflect() function.
        m_descriptors.insert(m_descriptors.end(), {
            GS_AudioSystemComponent::CreateDescriptor(),
            AudioDataAssetsSystemComponent::CreateDescriptor(),   <---------- this

that is where we tell the system about the descriptor

[vernal dove]

Yeah.

[red hearth]

I assume a breakpoint here triggers (at least, on the m_descriptors.insert line

and "knowing about it" causes it to reflect, so yoou also see a breakpoint in AudioDataAssetsSystemComponent::reflect

which is where it says it is for "Builders" (system component attribute)

[vernal dove]

Full Editor Build -> Run -> Debug Launch AssetBuilder.exe -> Breakpoints hit.

[red hearth]

and its not like, a problem with the attribute is it?

->Attribute(AZ::Edit::Attributes::SystemComponentTags, AZStd::vector<AZ::Crc32>({ "AssetBuilder" }))

vs what everything else has

->Attribute(AZ::Edit::Attributes::SystemComponentTags, AZStd::vector<AZ::Crc32>({ AZ_CRC_CE("AssetBuilder") }))

[vernal dove]

That would be so funny. Let me check.

[red hearth]

I mean, from that video, its reflecting it

and its giving it that attribute, which should cause it to be active in builders

being active in builders mean activate called on it

[vernal dove]

Changing the crc did make the asset appear in asset processor.

That might've done it.

[red hearth]

lol

[red hearth]

we should delete the constructor that allows AZ_CRC_CE to initializer list from a pointer

hat should pretty much fix everything

[vernal dove]

Double checked with the asset builder debug and yes it fires activate now.

Awesome.

[red hearth]

I'm actually not sure what happens when you do AZStd::vector<AZ::Crc32>({ "AssetBuilder" })

like... I assume that means that its feeding the char* string to AZ::Crc32's constructor

but clearly... it doens't happen right

[vernal dove]

Yeah... /shrug.

[red hearth]

so yeah everything was right but the attribute didn't make it

so the builder searching for anythiing tagged with that attribute didnt find it

so it didn't create the system component and activatae it

so it didn't register

[vernal dove]

Totally makes sense.

[red hearth]

now that it registers, its all taken care of

it should include dependnecies in the "Assets" view automatically

[vernal dove]

Yes! Now I just need to scrub all my asset regisytration system components and setregs.

[red hearth]

I've asked sig core to help me understand why the result is different

[vernal dove]

[red hearth]

yeah you dont need to have setregs to copy files n stuff

not if you're using this

it also compiles them in binary

so in your source folders they'll be xml files but in the shipped pak they'll be binary

thje file format for objectstream begins with a marker that indicates what it is, so when the Generic Asset Handler goes to read the file, it doesn't care if its the binary or xml format, it detects that from the first few bytes

so it is interchangeble

well at least we figured it out but I think I need to make some AZCore tests

becuase I expect the string literal to work

it shoudl be okay to construct an AZ::Crc32 with a string literal