#Converting Addressables to Entities at Runtime

1 messages · Page 1 of 1 (latest)

fringe parrot
#

Hello,

I currently have a system in place that allows me to download an Addressable catalog and convert a prefab inside of it into an Entity at Runtime.

However, now that I am done getting that to work, I am noticing in the Window->Entities->Archetypes window that my Chunks are being created very inefficiently.

The Addressables I built are very "atomic" compared to how people usually use them, so I typically ask for anywhere between 1 - 500+ Addressable catalogs to download->load into RAM->convert from GameObject prefab to Entity prefab.

This lets me "stream in a level" or a singular prop on demand. I have included a video to give a visual of what I am talking about. (For those curious, the terrain is not rendering with entities and remains a GameObject).

Is all of my chunk fragmentation caused by converting on-demand at arbitrary times? I am doing all my conversion the moment the Prefab is loaded in RAM on the main thread, which means each Prefab is being converted at different times.

Or is it caused by the fact I am iterating over a GameObject hierarchy and adding singular components on-demand causing lots of chunk swapping over time? (I am using an ECB to add the components, but am sometimes waiting a frame if the entity conversion takes too long and end up using multiple ECBs in the process).

I would build my own Archetype library to let me decide "Ah, this entity is going to end up as this Archetype when all is said and done", but some of the components being added are internal ones having to do with the LOD system or Rendering.

So because I have no ability to make sure those get added at the same time as my other components, does it even make sense to build myself an Archetype library to try and make sure all components I want to have on an Entity get added in one go?

#

I'll also include a screenshot of my Archetype window to show how inefficient the chunks are right now:

image.png
#

This is the largest Archetype I have that includes being a Prefab, as I convert the Addressables purely to Entity Prefabs and then spawn instances from those later.

tired osprey
#

shared components? are you trying to slapRenderMeshArray to your converted entites by any chance?

fringe parrot
#

I am, yeah. Or rather, RenderMeshUtility.AddComponents is when I pass my renderMeshArray into it.

#

Sorry, I cropped out the External Components part.

#

Also I guess RenderMeshUtility.AddComponentsis not using an ECB, it uses an entityManager directly.

tired osprey
#

RenderMeshArray is for baking only so you must drop it

rustic grove
#

Using ECB does not help with archetypes as ECB doesn't do any magic under the hood to optimize chunks. The main culprits are SharedComponents and individual AddComponent calls. Ideally, you should fully mitigate the use of shared components or add them in the very beginning of creating an entity.

For rendering you should use an overload that does not use RenderMeshArray as it is used during baking to optimize chunks, which is not really used during runtime and it just leads to more chunk fragmentation

For adding components you ideally should use CreateArchetype() method and just create entities from here. It will be much faster than adding individual componetns, as every AddComponent moves an entity from one chunk to another, leaving some never used chunks in the process

fringe parrot
#

Ok, but what about my mention of some components being internal and therefore I cannot add them to an Archetype of my own?

#

I suppose I just have to take the hit on those?

fringe parrot
#

It's weird. There are two overloads for RenderMeshUtility.AddComponents, one that explicity uses RenderMeshArray and one that does not.

The one that does not looks like it would not satisfy using RenderMeshUnmanaged either, because nothing sets the isInAuthoring to True. So it makes it feel like no mesh rendering component would be added at all.

Maybe these RenderMeshUtility.AddComponents methods are only meant to be used in Baking?

fringe parrot
rustic grove
fringe parrot
#

yeah, the asmref trick, right? i'd like to avoid it, of course, but it is what it is

rustic grove
fringe parrot
#

well yes, I see that part. But what I am saying is the comment in my red square says "RenderMeshArray is used at runtime"

#

or more specifically

# 
// At runtime all entities use RenderMeshArray.```
rustic grove
# 
            Entity entity,
            EntityManager entityManager,
            in RenderMeshDescription renderMeshDescription,
            MaterialMeshInfo materialMeshInfo)```
This is the correct overload
fringe parrot
#

False comments in their own code, wonderful

rustic grove
fringe parrot
#

yes, I know that overload does not

#

but I assumed it would need to add either RenderMeshArray OR RenderMeshUnmanaged

#

but it will add neither

#

but as you say, maybe neither is needed

#

the overload you pointed to does still add one sharedcomponent though

#

entityManager.SetSharedComponent(entity, renderMeshDescription.FilterSettings);

rustic grove
#

Because it is only used to optimally combine meshes and materials. The overload I sent requires you to register the meshes and materials by yourself

rustic grove
rustic grove
#

At least you have one shared component less

fringe parrot
#

ok, well, the less shared components I have the less fragmentation I'll get?

rustic grove
#

Yes

fringe parrot
#

ok

#

alright thanks, I'll convert over to using this and see what happens before moving as much to a "set the archetype all at once" as I can in the future.

#

just wanted to make sure whatever I do is "worth doing" before trying

rustic grove
fringe parrot
#

So instead of using a RenderMeshArray to make my MaterialMeshInfo, I have to register the mesh with some other more global array contained somewhere in the Entities Graphics library?

#

I see. There are RegisterMaterial and RegisterMesh methods

rustic grove
fringe parrot
#

dang it

rustic grove
#

(E.g. know what you register by storing in a hashset for example)

#

It is really simple

fringe parrot
#

you're right, thanks for the heads up

#

i was going to assume it would do that for me

dim patrol
#

I know this is not the topic but any reason you don't just ship your entities as entities with content management?

#

You have to build your game objects as addressables what, it's the same process to bake and create a content bundle

#

(Apart from a complete lack of documentation)

fringe parrot
#

content management does not produce bundles that share by reference when you build them (at least not in the way I want them to)

#

i am building my addressable catalogs on a per-prefab basis (catalog for tree, catalog for house, etc.)

#

they are being built as individual bundles too, so when a tree and a house share the same texture (for example), it is not being duplicated

#

if I were to use content management to build a house and then later a tree, as far as I know it produces one bundle with everything in it, and thus they would be copies and not references

#

in other words, addressables is more mature and featured (again, as far as I know)

dim patrol
#

addressables does the same thing by default

#

it puts copies in each bundle

fringe parrot
#

yes, i am not using addressables by default

dim patrol
#

it only deduplicates if you specifically put an asset that is shared into its own bundle

#

this duplicate is actually often a preferred behaviour which is why its the default behaviour

#

otherwise it causes memory to never release

fringe parrot
#

i am using addressables in such a way that I am producing no duplicates (except the damn "error material/shader" or whatever)

dim patrol
#

so while duplication might increase app file size, it can actually reduce memory usage by allowing assets to unload

fringe parrot
#

in any case, this is why i am not using content management

dim patrol
#

but btw, content manage does support deduplication

fringe parrot
#

well if it does, I have no idea how to use it. the documentation I have seen is that you just pass in the subscene you want and it makes a bundle for it

#

and so if I have a subscene for my house, it makes a bundle for that

#

and then a subscene for my tree, it makes a bundle for that

dim patrol
#

yeah it wont for that case, at least on disk

fringe parrot
#

i am using addressables in a weird way, you might say

dim patrol
#

but if you load it at runtime it'll merge catalogs

#

and ignore the second tree

fringe parrot
#

ah, well, I don't want to download unnecessary data either

#

with addressables, if the house and tree share a texture, it will only download it once and then used the cached version

#

(my texture is in its own bundle by itself)

dim patrol
#

are you doing this for modding or for the sake of runtime updates by you only?

fringe parrot
#

the second one

dim patrol
#

because if it's just you doing runtime updates it won't have to do this

#

while it doesn't support incremental builds

fringe parrot
#

i don't use addressable's incremental builds

dim patrol
#

it just rebuilds the entire catalog but will only pull new files

fringe parrot
#

i find those are for people who make DLC packages

#

if content management can build each one of my assets into its own discrete bundle, I would consider exploring it

dim patrol
#

because yeah i find content management actually super efficient for just pushing new content, it checks every asset individually and only downloads what it's missing in your new build

dim patrol
#

i just spent 3 days over 2 weekends figuring out how to load arbitrary catalogs

#

so i could load mods

fringe parrot
#

what I am doing warrants its own blog post, but I have 1 addressable catalog per prefab, basically

dim patrol
#

but for this specific case it'll build duplicate assets unfortunately, just wont load it

#

if you're the only one building content though, you're better off just building the entire catalog each time and pushing a new version of it for every small update

fringe parrot
#

so if I want a tree I've never seen before, I just download the catalog for that and load it, and addressables either uses cached assets it already has or downloads new ones

#

my goal is to also have small catalogs

#

so if you want to update a tree or a rock, you don't have to dirty whatever massive catalog you might have and require downloading it again

#

you just dirty the small one for that tree or rock or whatever

dim patrol
#

the only pain is uploading the whole thing again

fringe parrot
#

right, i meant the catalog file itself

#

the manifest

#

or whatever, i don't know what content management calls it

#

if you had like 10,000 items in it, it becomes kinda large in itself

#

believe me, I wish I could use content management

#

I'd love to pre-bake these entities

#

I've wondered if I could just use the entity cache part of content management throw out their bundles and continue to use addressables to load the assets into memory

#

it'd be pretty easy for me to say "Addressables, load the assets for this tree, and when you're done, load the binary data for the entity that would use them"

dim patrol
#

why can't you do that?

#

if you really wanted you can just store addressable keys on the entities for their assets

#

i do something similar but just weakassetreferences

#

i autoconvert materialmeshinfo into weakassetreferences during baking (but you could do it to addressable key instead)

#

and have a system to manually load assets when in range

fringe parrot
#

well how do I load the baked entity data at runtime? with the content management stuff? I just haven't tried yet.

dim patrol
#

yeah

#

the pain would be though managing 2 separate bundle pipelines

fringe parrot
#

that's probably not a big deal for me to just slap it into my existing pipeline as an extra step

#

I'll explore it if you're giving a endorsement

dim patrol
#

i'm not going to say it's not a bunch of work

#

but doing runtime conversion also seems like a bunch of work to me ;'(

fringe parrot
#

well I have already done most of the runtime conversion, but it uses precious milliseconds of frametime

#

and it may not scale well

dim patrol
#

maybe the secret to runtime conversion is going back to entities 0.51! thinksmart

#

but if you've done the work probably shouldn't be reworking everything

fringe parrot
#

i'm not doing this for anyone but myself, so time is infinite

dim patrol
#

i was more just curious about the choice to use addressables

fringe parrot
#

ah

#

well like I said, I would love to not have to convert entities at all as I know it would be faster to just load them from a file pre-baked

#

and they would be less prone to errors

#

and I could have Authoring components actually work ahead of time, if I ever decide to add any

#

right now my converter just ignores any authoring components (I don't have any anyway)

#

i assume the entities won't automatically hook up the assets since I'm loading them from Addressables and it'll be up to me to slide the materials and meshes into those weakassetreferences you mentioned

#

though I guess Graphics could be Entity Graphics lul

fringe parrot
dim patrol
#

i basically just swap out the unityobjectref<T> on materialmeshinfo and replace them with null

#

and store a weakassetreference to the assets on my own component

#

and then load the assets in proximity and write them to mmi

#

this way i still bake the full component set required for entities graphics

#

but leave out asset references so they aren't loaded when loading the subscene

fringe parrot
#

so you keep the entities floating around and just shut off the rendering that way based on proximity?

dim patrol
#

yeah

#

i simulate the whole world all the times

fringe parrot
#

makes sense

dim patrol
#

but i don't the whole worlds assets loaded all time

fringe parrot
#

does that work with submeshes and such too? i assume rendermesharrays are baked and whatnot.

dim patrol
#

no, i specifically can't do this with submeshes

#

i don't use submeshes though

fringe parrot
#

ah, damn

dim patrol
#

submeshes require rendermesharray

#

and i don't want to use rma

fringe parrot
#

ah I see, didn't know they required it

#

i was hoping to support any arbitrary gameobject, which sometimes includes submeshes

#

though I suppose it's not the end of the world to tell artists to avoid them

dim patrol
#

you can use entities graphics without rendermesharray if you dont use submeshes

#

but yeah it requires rma atm for submeshes unfortunately

#

basically uses sequential elements in this array for rendering each mesh

fringe parrot
#

it's only multiple materials that causes sub-meshes right?

#

i forget

dim patrol
#

yes

fringe parrot
#

ok, well that's not so bad

dim patrol
#

well technically youc an have submeshes and a single material reused across all of them but thats pretty silly

fringe parrot
#

probably could split up into "real meshes" in my pipeline if I really cared

dim patrol
# 
        private static void AddMaterialMeshPresentation(
            EntityCommandBuffer ecb, Entity entity, RefRW<MaterialMeshInfo> materialMeshInfo, RefRO<RenderMeshUnmanaged> rm)
        {
            var mmp = new MaterialMeshPresentation();
            mmp.Mesh.SetId(rm.ValueRO.mesh.ToUntypedWeakReferenceId());
            mmp.Material.SetId(rm.ValueRO.materialForSubMesh.ToUntypedWeakReferenceId());

            ecb.AddComponent(entity, mmp);
            ecb.SetComponentEnabled<MaterialMeshPresentation>(entity, false);

            materialMeshInfo.ValueRW = new MaterialMeshInfo(default, default, rm.ValueRO.subMeshInfo.SubMesh);
        }```

but this is basically what i'm doing
#

in a baking system

fringe parrot
#

is MMP your own component?

dim patrol
#

yes

#

oh i allow submeshes if they're created as separate entities

fringe parrot
#

ah

#

and what decides that? the baker?

dim patrol
#

which is really just each entity only actually renders 1 mesh

fringe parrot
#

right

#

like how would you tell the baker to never use render mesh array?

dim patrol
#

if you define ENABLE_MESH_RENDERER_SUBMESH_DATA_SHARING or not

fringe parrot
#

oh

dim patrol
#

the historical way was that unity would create each submesh as a separate child entity

#

they added support for a single entity to render all submeshes

#

which sounds great (and kind of is for a lot of cases)

#

but requires RMA

#

and isn't really easy to swap materials etc

fringe parrot
#

because you'd have to remake the array every time

dim patrol
#

yeah managed and a shared component

#

anyway i've always just prefered to avoid submeshes if possible

# 
            // Single mesh object
            foreach (var (materialMeshInfo, renderMesh, materialMeshInfoEnabled, entity) in SystemAPI
                .Query<RefRW<MaterialMeshInfo>, RefRO<RenderMeshUnmanaged>, EnabledRefRW<MaterialMeshInfo>>()
                .WithAll<MaterialMeshPresentationBaking>()
                .WithEntityAccess()
                .WithOptions(EntityQueryOptions.IncludePrefab | EntityQueryOptions.IncludeDisabledEntities | EntityQueryOptions.IgnoreComponentEnabledState))
            {
                if (materialMeshInfo.ValueRO.HasMaterialMeshIndexRange)
                {
                    Debug.LogWarning("Using MaterialMeshPresentationAuthoring on MultiMesh entity");
                    return;
                }

                AddMaterialMeshPresentation(ecb, entity, materialMeshInfo, renderMesh);
                materialMeshInfoEnabled.ValueRW = false;
            }

            var mask = SystemAPI.QueryBuilder().WithAll<MaterialMeshInfo, RenderMeshUnmanaged>().Build().GetEntityQueryMask();

            // Multi mesh object
            foreach (var (additionalEntities, entity) in SystemAPI
                .Query<DynamicBuffer<AdditionalEntitiesBakingData>>()
                .WithNone<MaterialMeshInfo>()
                .WithAll<MaterialMeshPresentationBaking>()
                .WithEntityAccess()
                .WithOptions(EntityQueryOptions.IncludePrefab | EntityQueryOptions.IncludeDisabledEntities | EntityQueryOptions.IgnoreComponentEnabledState))```

but i do support it, in the baking
fringe parrot
#

well I thought the way to avoid submeshes on gameobjects was to also just make whatever submesh you might want onto its own renderer (aka, separate gameobject)

dim patrol
#

submeshes are basically just the same thing as having 2 separate renderers

#

they're individual draw calls

fringe parrot
#

yeah

#

alright, well you've given me a lot to think about. I'll run some experiments with building a sub-scene that contains a single prefab using content management and see how that goes

dim patrol
#

btw there is EntitySceneReference

#

which is basically a single prefab gameobject built as a standalone subscene

#

oops wrong one sorry

#

EntityPrefabReference

fringe parrot
#

hm, when is it used? I see there is a constructor that takes in a GameObject

#

PrefabReference sample

This sample demonstrates how to use an EntityPrefabReference. Whereas directly baking a prefab in each SubScene creates one baked entity per SubScene, an EntityPrefabReference allows multiple SubScenes to reference a single baked prefab entity.

#

Ah, well, I don't think I'm going to have any sub-scenes with a duplicate prefab in them.

#

I'm not building any levels in the editor by hand in the traditional sense.

#

But if I can pass an EntityPrefabReference to Content Management and it treats it as a sub-scene, that sounds nice

#

means I don't have to worry about doing extra steps with sub-scene management during building

fringe parrot
#

Looks like simply stopping use of RenderMeshArray did significant work

image.png
green summit
#

And I thought RenderMeshArray was such a PITA. Now I can say I'm even more convinced of that after reading this thread

#

Also MaterialMeshInfo

#

FWIW those are on the way out in the short/mid term (around 6.5-6.6). Hopefully I can get sign-off to give more details soon

crude elm
green summit
#

no one truly enjoys working with the current entities graphics code/framework/api right now
This is more than fair I think

#

And yes the EG API will basically get almost a full rework

#

We'll collect feedback from you as well to make sure we aren't missing anything crucial (if you're interested of course). I would like to do it here and/or on the forums. Hopefully I have some updates soon

crude elm
crude halo
#

I wanted to ask about all this stuff during the last office hours but felt like it was kind of "are we there yet" and didnt, but very cool to hear progress happening

green summit