Trouble using Content Management to load subscenes from mods at runtime | Official Unity Discord | Page 1

near rain Sep 29, 2023, 11:10 PM

#

Modders currently have a template project they work from to create new systems and components. I am adding support for them to add new entities to the world. I am using the Content Management system to publish the subscene data, which contains their new entity prefabs, like this (which came from this thread: https://forum.unity.com/threads/how-to-actually-use-enable_content_delivery-in-build.1495526/):

     [MenuItem("Modding/Content Update (fixed)")]
    static void Execute()
    {
      string buildFolder = Path.Combine(Application.streamingAssetsPath, "ContentBuild");
      if (!string.IsNullOrEmpty(buildFolder))
      {
        var buildTarget = EditorUserBuildSettings.activeBuildTarget;

        if (!Directory.Exists(Path.Combine(Path.GetDirectoryName(Application.dataPath),
              $"Library/ContentUpdateBuildDir")))
          Directory.CreateDirectory(Path.Combine(Path.GetDirectoryName(Application.dataPath),
            $"Library/ContentUpdateBuildDir"));
        if (!Directory.Exists(Path.Combine(Path.GetDirectoryName(Application.dataPath),
              $"Library/ContentUpdateBuildDir/{PlayerSettings.productName}")))
          Directory.CreateDirectory(Path.Combine(Path.GetDirectoryName(Application.dataPath),
            $"Library/ContentUpdateBuildDir/{PlayerSettings.productName}"));

        var tmpBuildFolder = Path.Combine(Path.GetDirectoryName(Application.dataPath),
          $"Library/ContentUpdateBuildDir/{PlayerSettings.productName}");

        var instance = DotsGlobalSettings.Instance;
        var playerGuid = instance.GetPlayerType() == DotsGlobalSettings.PlayerType.Client
          ? instance.GetClientGUID()
          : instance.GetServerGUID();
        if (!playerGuid.IsValid)
          throw new Exception("Invalid Player GUID");

        var subSceneGuids = new HashSet<Unity.Entities.Hash128>();
        for (int i = 0; i < EditorBuildSettings.scenes.Length; i++)
        {
          var ssGuids = EditorEntityScenes.GetSubScenes(EditorBuildSettings.scenes[i].guid);
          foreach (var ss in ssGuids)
          {
            if (!subSceneGuids.Contains(ss))
            {
              Debug.Log("GUID -> " + ss);
              subSceneGuids.Add(ss);
            }
          }
        }
        if (subSceneGuids.Count == 0)
        {
          Debug.LogError("No SubScenes found");
          return;
        }
        RemoteContentCatalogBuildUtility.BuildContent(subSceneGuids, playerGuid, buildTarget, tmpBuildFolder);

        var publishFolder = Path.Combine(Path.GetDirectoryName(Application.dataPath), "Builds",
          $"{buildFolder}-RemoteContent");
        RemoteContentCatalogBuildUtility.PublishContent(tmpBuildFolder, publishFolder, f => new string[] {"all"});
        Debug.Log($"Publish Content Update complete to path {publishFolder}");
      }

The resulting content is packaged up with the modder's other DLL's, and uploaded to the steam workshop. My game consumes that folder for subscribed mods, and then tries to load the subscene data like this:

      RuntimeContentSystem.LoadContentCatalog(null, ModLoader.ModContentCache, "all");
      ContentDeliveryGlobalState.Initialize(null, ModLoader.ModContentCache,
        "all", s =>
        {
          if (s >= ContentDeliveryGlobalState.ContentUpdateState.ContentReady)
          {
            PerformWholeGameInitializationProcess();
          }
        });

I think I'm missing a step though because although it appears the Initialize was successful, the modded subscene does not actually get created and put into the game. The files in the ModContentCache path look correct, they are the result of the publish operation. I've attached an image.

Any ideas on what else needs to be done to actually load the subscene?

sharp perch Sep 30, 2023, 12:36 AM

#

Maybe I'm missing something because I'm on a phone but I don't actually see where you're actually loading the subscene into a world

#

I.e. Calling something like SubSceneSystem.LoadSceneAsync

near rain Oct 1, 2023, 2:08 PM

#

sharp perch Maybe I'm missing something because I'm on a phone but I don't actually see wher...

Yeah this is the part I'm confused about. How do I actually get the subscene reference that is ostensibly in the content archive? There are docs on loading weakly referenced scenes at runtime, but I'm not sure how to access those references in-game, seeing as they were created and baked in the modding project, and presumably live in the modded project's entities subscene, which is the very thing I'm trying to load

shell monolith Oct 1, 2023, 2:44 PM

#

near rain Yeah this is the part I'm confused about. How do I actually get the subscene ref...

I think you can pass guid via network

#

load content archive

#

and use that guid afterwards

#

I'm unfamiliar with process as I never did it though

near rain Oct 1, 2023, 2:45 PM

#

shell monolith I think you can pass guid via network

How can I access the guid? And what do you mean via network?

shell monolith Oct 1, 2023, 2:46 PM

#

near rain How can I access the guid? And what do you mean via network?

guid of scene asset

#

it matches subscene guid

#

via network I guess - from some asset maybe?

#

your mods must provide it

#

for example it can be auto generated

near rain Oct 1, 2023, 2:48 PM

#

ah so maybe I can store the scene guid to some file when I publish, then read that guid at runtime in my game?

#

it seems convoluted but it might work

shell monolith Oct 1, 2023, 2:49 PM

#

near rain ah so maybe I can store the scene guid to some file when I publish, then read th...

yes

near rain Oct 1, 2023, 3:53 PM

#

shell monolith yes

OK I think I'm getting closer. I have this now:

      ContentDeliveryGlobalState.Initialize(null, ModLoader.ModContentCache,
        "all", s =>
        {
          if (s >= ContentDeliveryGlobalState.ContentUpdateState.ContentReady)
          {
            foreach (var modInfo in ModLoader.ModInfo.Values)
            {
              var subceneGuid = modInfo.SubsceneGuid;
              Debug.Log($"Loading subscene for mod {modInfo.FullName} with guid {subceneGuid}");
              SceneSystem.LoadSceneAsync(Ecs.World.Unmanaged, new Hash128(subceneGuid));
            }
            StartCoroutine(LoadGameAfterSubsceneLoads());
          }
        });

The subscene guid looks correct. However, in LoadEntitySceneAsync it queries for SceneReference components and does not find my subscene, and it doesn't appear that my subscene actually gets loaded

#

mod content cache looks like this

shell monolith Oct 1, 2023, 4:15 PM

#

near rain OK I think I'm getting closer. I have this now: ``` ContentDeliveryGlobal...

LoadSceneAsync creates an entity if it doesn't find one

#

check entity it returns

#

and see if it's valid

near rain Oct 1, 2023, 4:15 PM

#

yea it hits this line

            if (sceneEntity == Entity.Null)
                return CreateSceneEntity(world, sceneGUID, parameters);

but none of the entities from my subscene are actually around

shell monolith Oct 1, 2023, 4:15 PM

#

check entity it returns

#

and inspect that entity

#

this entity is a handle to subscene loading

#

it stores component with information about loading status

near rain Oct 1, 2023, 4:23 PM

#

are you referring to RequestSceneLoaded? Its LoadFlag is just set to 0 atm, which actually doesn't correspond to anything on the enum.

near rain Oct 1, 2023, 4:23 PM

#

shell monolith it stores component with information about loading status

is there some other step to actually load this that I'm missing?

shell monolith Oct 1, 2023, 4:24 PM

#

near rain is there some other step to actually load this that I'm missing?

nah, just calling this method should be it

#

just find this entity in hierarchy

#

and look at inspector

shell monolith Oct 1, 2023, 4:25 PM

#

near rain are you referring to `RequestSceneLoaded`? Its LoadFlag is just set to 0 atm, wh...

no, RequestSceneLoaded is what gets added by LoadSceneAsync

#

this is a trigger to load it

#

and 0 just means, it's a normal subscene load

#

without any options

near rain Oct 1, 2023, 4:26 PM

#

ok yea i do actually see it in the inspector

(second one)

shell monolith Oct 1, 2023, 4:26 PM

#

no

#

that's not what I mean

#

what I mean is find entity itself

#

it's literally an entity you can inspect

near rain Oct 1, 2023, 4:28 PM

#

this should be it

shell monolith Oct 1, 2023, 4:28 PM

#

potentially

#

it's empty

#

but it's loaded

#

huh?

near rain Oct 1, 2023, 4:32 PM

#

It should have at least 1 entity in it (the placeholder entity defined in the mod subscene)

#

Well I'm at a loss. I tried setting the load flags manually but it doesnt seem to matter:

              var sceneEntity = SceneSystem
                .LoadSceneAsync(Ecs.World.Unmanaged, new Hash128(subceneGuid), new SceneSystem.LoadParameters
                {
                  Flags = SceneLoadFlags.NewInstance,
                });

#

The scene entity exists but nothing is in it

#

Do I need to weakly reference the scene in my mod or something? I have to be missing a step

shell monolith Oct 1, 2023, 4:40 PM

#

near rain Do I need to weakly reference the scene in my mod or something? I have to be mis...

no, your mod should just bake subscene into entity asset

near rain Oct 1, 2023, 4:42 PM

#

this is the only baker in the modded subscene. It loads some configs which reference prefabs that will become entities, and stores those entity prefabs in a DynamicBuffer. So there should be, in this case, an entity that this baker is running on, and 1 other entity prefab that gets created during this baking process.

  public class ModdedEntityPrefabBaker : Baker<ModdedEntityPrefabBakerAuthoring>
  {
    public override void Bake(ModdedEntityPrefabBakerAuthoring authoring)
    {
      Debug.Log("Baking modded entities...");
      var loader = (UserModLoader)Activator.CreateInstance(typeof(UserModLoader));
      var configs = loader.DefineEntityConfigs();
  
      var entityMetaData = AddBuffer<EntityMetaDataElement>(GetEntity(TransformUsageFlags.Dynamic));

      for (var index = 0; index < configs.Count; index++)
      {
        var config = configs[index];
        var entity = GetEntity(authoring.Prefabs[index], TransformUsageFlags.Dynamic);
        var entityMetaDataElement = new EntityMetaDataElement
        {
          Entity = entity,
          Guid = config.EntityReferenceGuid.ToString()
        };

        entityMetaData.Add(entityMetaDataElement);
      }
    }
  }

If I debug the baker it seems to do everything correctly. The entities are in the modded scene in the mod project just fine

shell monolith Oct 1, 2023, 4:54 PM

#

is size of subscene calculated correctly?

shell monolith Oct 1, 2023, 4:54 PM

#

near rain this should be it

here

near rain Oct 1, 2023, 4:55 PM

#

It really just seems like its creating an empty scene entity with the guid i'm giving it, it's not actually loading my scene

shell monolith Oct 1, 2023, 4:55 PM

#

but is size correct or not?

near rain Oct 1, 2023, 4:56 PM

#

i'm not sure what to compare it with. The scene data in my modded project is like 104kb, which does not match the File Size

shell monolith Oct 1, 2023, 4:56 PM

#

also: this baker supposed to be entity inside of subscene you load, right?

shell monolith Oct 1, 2023, 4:57 PM

#

near rain i'm not sure what to compare it with. The scene data in my modded project is lik...

check files in modded folder

near rain Oct 1, 2023, 4:57 PM

#

shell monolith also: this baker supposed to be entity inside of subscene you load, right?

correct

shell monolith Oct 1, 2023, 4:57 PM

#

it must have baked subscenes somehwere

#

if I'm not mistaken it's name will be guid

#

but I could be

near rain Oct 1, 2023, 4:57 PM

#

This is the content that gets published that I"m trying to load. I'm not sure what part of it is the subscene

shell monolith Oct 1, 2023, 4:57 PM

#

just look if there's a file 17kb somehwere

near rain Oct 1, 2023, 5:00 PM

#

there isn't, but in the 8a folder I can see the entities subscene binary from the modded subscene

shell monolith Oct 1, 2023, 5:01 PM

#

near rain there isn't, but in the 8a folder I can see the entities subscene binary from th...

can you show it?

near rain Oct 1, 2023, 5:02 PM

#

First image is the 8a folder, second image is just the .entities file itself from the built modded unity project. The files look the same

#

shell monolith Oct 1, 2023, 5:02 PM

#

is guid matching?

near rain Oct 1, 2023, 5:03 PM

#

it does not appear so. If I manually use the other file name as the Guid it still does not work though

shell monolith Oct 1, 2023, 5:03 PM

#

try to load using this name

#

as guid

near rain Oct 1, 2023, 5:04 PM

#

I tried using b2844cf989ef6ed40b7ddf0a13b92591 and 8af87a38755f7c9a6514b1d80e73cac5 the result is the same: a seemingly empty subscene

shell monolith Oct 1, 2023, 5:04 PM

#

then I wonder whether content is actually loaded

near rain Oct 1, 2023, 5:05 PM

#

      ContentDeliveryGlobalState.Initialize(null, ModLoader.ModContentCache,
        "all", s =>
        {
          if (s >= ContentDeliveryGlobalState.ContentUpdateState.ContentReady)
          {
            foreach (var modInfo in ModLoader.ModInfo.Values)
            {
              var subceneGuid = "8af87a38755f7c9a6514b1d80e73cac5";
              Debug.Log($"Loading subscene for mod {modInfo.FullName} with guid {subceneGuid}");
              var sceneEntity = SceneSystem
                .LoadSceneAsync(Ecs.World.Unmanaged, new Hash128(subceneGuid));
            }

I get inside that if block so according to the ContentDelivery code it should be loaded

near rain Oct 2, 2023, 2:24 PM

#

Ok there's a couple problems that we've fixed so far. The confusing part is how the remoteUrl is used. My intention was not to use a remote URL, since the mod is local. The docs say to not include the remoteUrl argument if you want to use local files, but the catch is: it hardcodes using the Streaming Assets folder for local files. But you aren't supposed to write to the local assets folder. So there's no way to specify a local folder to load a part from that. This seems like a huge gap in the implementation that Unity should fix.

We hosted our files on a local webserver to test the downloading from ContentDeliveryGlobalState.Initialize using a remote url and it does download the files into the appropriate folder now, but still no luck at all actually loading the subscene. And long term it's not really viable at all serve this content from a remote server, we're just doing that right now to try to get this working.

The fundamental process of loading a subscene from another project seems poorly supported, and thus modding for entities is pretty difficult to setup.

near rain Oct 3, 2023, 5:46 PM

#

@sharp perch I hate to poke you but you said you got this working and made it seem like it was an easy process. We've been banging our heads against this problem for days now.

Fundamentally we need to load a subscene that ultimately comes from another local Unity project at runtime for modding. We've gone as far as to modifying the Content Management source code and the Entities.Scenes source code to at least try to hack it to make it load the external scene. We've tried loading from the content catalog and also hacking to load the StreamingAssets content from the other project with no success.

Are you sure you are actually loading scenes from other unity projects? We just be missing something critical having spent so many hours trying to figure this out.

The current state of the issue:

In the Modded unity project, we have a editor script that builds and publishes the streaming assets using The RemoteContentCatalogBuildUtility. This seems to yield the expected files (a bunch of 2 letter folders and a catalogs.bin file). Stepping through that code I can see it serializing the data I'd expect from the subscene.
We've pushed this content onto a webserver so we can give the ContentDeliveryGlobalState.Initialize a remote URL to work from (even though we just want to be able to give that function a file path, not a URL, but we'll have to address this later I guess)
In the actual Game's code, we created a new ModdedSceneLoaderSystem that does this in OnUpdate (which we disable after it does it once):

      ContentDeliveryGlobalState.Initialize(modUrl, ModLoader.ModContentCache,
        "all", s =>
        {
          if (s >= ContentDeliveryGlobalState.ContentUpdateState.ContentReady)
          {
            foreach (var modInfo in ModLoader.ModInfo.Values)
            {
              var subsceneGuid = modInfo.SubsceneGuid;
              Debug.Log($"Loading subscene for mod {modInfo.FullName} with guid {subsceneGuid}");
              SceneSystem.LoadSceneAsync(Ecs.World.Unmanaged, new Hash128(subsceneGuid));
            }
          }
        });

Although it appears the content from the URL gets copied into the ModContentCache folder, and a new subscene does get created, it does not appear to have the entities that were baked into it from the mod.

Also important to note behavior differs between editor and builds. The editor doesn't ever get the callback that the content is ready, even though it downloads correctly. In the build it gets the signal that the content is ready, but the subscene has nothing in it; furthermore the main subscene from the actual game does not appear to have its baked entities either (but in the editor its fine). It's a big mess!

We've really tried so much more than this but I'm trying to keep things concise.

sharp perch Oct 3, 2023, 6:07 PM

#

I've never tried to do remote subscenes, I simply got support for subscenes loaded at runtime to override existing data.
I've intentional not investigated this much as I've read a lot of issues users are having and I don't have the time atm and my timeline for needing this is > 6 months so I'm happy to let it settle / fix.

#

@devout wraith has a lot of open bug reports related to this

#

Some 3-6 months old, not sure on the status

near rain Oct 3, 2023, 6:14 PM

#

Ok at least it wasn't something obvious that we're missing. I guess we'll keep trying to hack at it. Unfortunately this feature seems extremely poorly supported, I might just have to find a different way outside of the Content Management api

#

It seems absolutely critical that there is a solid API for this for DOTS modding

shell monolith Oct 3, 2023, 6:18 PM

#

near rain It seems absolutely critical that there is a solid API for this for DOTS modding

not just modding, it's pretty crucial for online games as well

#

loading assets at runtime for example

sharp perch Oct 3, 2023, 6:18 PM

#

I'm not sure about official modding but remote bundles are critical

#

Yeah what issue said

near rain Oct 3, 2023, 6:19 PM

#

In the mean time I would settle for the ability to convert gameobjects to entities at runtime again 😦

shell monolith Oct 3, 2023, 6:31 PM

#

near rain In the mean time I would settle for the ability to convert gameobjects to entiti...

I'd rather wait on Unity fix this

#

don't want to be invested into awful solution if Unity will fix it in a couple months

devout wraith Oct 4, 2023, 3:21 AM

#

near rain <@133111808562036737> I hate to poke you but you said you got this working and m...

Have u successfully make it work at ur main game project instead of another new unity project for modding?

near rain Oct 4, 2023, 9:57 PM

#

devout wraith Have u successfully make it work at ur main game project instead of another new ...

I haven't tried. I just ended up writing a minimal Game Object to Entity runtime converter and load from asset bundles. Super simple and bypasses all of this Content Management stuff that clearly is not ready for primetime

#Trouble using Content Management to load subscenes from mods at runtime