#Threading

1 messages · Page 1 of 1 (latest)

slate wave
shadow magnet
#

so what does Load do?

slate wave
#

reads a proprietary bundle file and extract things from there

#

plain c# code

shadow magnet
#

how big is the bundle?

slate wave
#

around 2gbs

shadow magnet
#

well theres your problem lol

slate wave
#

why's that? we don't read everything at once

shadow magnet
#

oh, then how big is the texture?

slate wave
#

i doubt more than 40kb-ish each?

#

i believe the problem is because we're iterating on an array of like 1403 objects and each object is locking the thread waiting for the texture to load

shadow magnet
#

hrm, you should be able to stick them into threads then, have you tried just using the Thread class?

slate wave
#

only from System.Threading.Tasks

# 
 private async Task CompileModels(string mapname, RSM[] objects, Action<string, string, object> callback) {
            List<Task<RSM.CompiledModel>> tasks = new List<Task<RSM.CompiledModel>>();

            foreach (var model in objects) {
                var t = Task.Run(() => {
                    var compiledModel = ModelLoader.Compile(model);
                    LoadModelTexture(compiledModel);
                    return compiledModel;
                });
                tasks.Add(t);
            }

            RSM.CompiledModel[] compiledModels = await Task.WhenAll(tasks);
            callback.Invoke(mapname, "MAP_MODELS", compiledModels);

        }

        private void LoadModelTexture(RSM.CompiledModel model) {
            HashSet<string> textures = new HashSet<string>();

            foreach (var nodeMesh in model.nodesData) {
                //load its textures
                foreach (var textureId in nodeMesh.Keys) {
                    var texture = "data/texture/" + model.rsm.textures[textureId];
                    //load texture
                    if (!textures.Contains(texture)) {
                        textures.Add(texture);
                        FileManager.Load(texture);
                    }

                    if (textures.Count == model.rsm.textures.Length) {
                        break;
                    }
                }
            }
        }
#

but I'm getting the unfamous UnityException

shadow magnet
#

what does it say?

slate wave
# 
UnityException: SupportsTextureFormatNative can only be called from the main thread.
Constructors and field initializers will be executed from the loading thread when loading a scene.
Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.
#

triggered from the line FileManager.Load(texture)

azure moth
#

Unity's objects aren't thread safe, so don't try to create or manipulate them outside of the main thread

#

You said it's loading from a 2gb bundle, but this code loads a single file. Where is your time being spent?

slate wave
#

yeah, I'm wondering if you guys would know a workaround of not locking the thread / taking less time to load that thing

#

the 2gb file is read at startup, we then cache its file descriptors

#

when we try to access any file inside it, we just hit the descriptors hashset and get the byte array of the file

shadow magnet
#

you'll need to figure out what method in Load can't be used in the main thread

slate wave
#

the time consuming part is at mr.material.mainTexture = FileManager.Load("data/texture/" + textureFile) as Texture2D;

slate wave
# shadow magnet you'll need to figure out what method in Load can't be used in the main thread

it narrows down to

 private static Texture2D toUnityTexture(object texture) {
            if (texture is Texture2D) {
                return texture as Texture2D;
            }

            Texture2D t = null;

            if (texture is FileManager.RawImage) {
                t = new Texture2D(0, 0);
                t.LoadImage(((FileManager.RawImage) texture).data);
            } else if (texture is BMPImage) {
                t = ((BMPImage) texture).ToTexture2D();
            } else if (texture is TGALoader.TGAImage) {
                t = ((TGALoader.TGAImage) texture).ToTexture2D();
            }

            return t;
        }
#

which is inside FileCache.cs

azure moth
#

I mean, what part of Load? Maybe something is much slower than it should be: grf.GetDescriptor, grs.GetData, creating a memory stream, etc

slate wave
#

it's at the rendering part, we are past the bundle loading at this moment

slate wave
azure moth
#

yes, but look at all the work FileManager.Load is doing. Figure out what the most time consuming part of that work is

slate wave
#

I'm wondering if I assign a custom component to each of these models and tell them which is their texture and let them load at their own pace would make it faster?

shadow magnet
shadow magnet
slate wave
#

gonna take a look at that, thank you!

#

Would Resources.LoadAsync help in this case?

shadow magnet
#

it should

#

if you're loading from the resources folder at least

slate wave
#

Yea, I can write a tool to dump textures from the bundle file to the resources folder

shadow magnet
#

at runtime or in the project?

raw cargo
#

you probably won’t benefit much from offloading to another thread. So you can use coroutines or async/await without Threadpool.

#

or if you run everything on a background thread, create a pub/sub queue of texture byte[] to load into actual textures, and pop only a few items off that queue each frame until empty

slate wave
slate wave
#

I was thinking about asset bundles or extracting the textures to resources and use the load async methods… or maybe so I don’t lock the thread I could build all meshes without textures and then just load their textures in another situation

slate wave
#

I tried enqueuing like this TexturesQueue.Enqueue(new KeyValuePair<MeshRenderer, string>(mr, "data/texture/" + textureFile)); and then after loading the meshes I dequeue on a coroutine and load them waiting for the end of the frame... However the queue is emptied but not all textures are loaded... any clues why?

raw cargo
#

you want to do all texture loading from file/url incrementally such that the upload of the texture to the GPU via Texture2D.LoadImage() or .Apply() does not happen for too many images all in one frame because this blocks the main thread and you can’t yet do anything about it. The reading of the file/remote data byte[] can be handled async in any way you like (other thread, coroutine, …) as it doesn’t touch unity objects yet. Same thing for meshes. Any delays are caused by mesh uploads to the gpu, all basic data processing and mesh construction on the cpu can be offloaded to a thread again.

slate wave
#

I see

raw cargo
#

If you load prefabs I.e. via addressables or scenes normally that contain lots of of objects… still the loading can happen in the background, but at some point unity will iterate over all objects to initialize its internal caches and references… this will also block the main thread and nothing can be done about it

#

unless you have a source license

slate wave
#

makes sense... at this moment we have the map loading being done using coroutines and we dispatch a WaitForEndOfFrame at every batch size... this has helped us to prevent the editor/client from freezing. However it is slower than the original client which does things using opengl

#

using coroutines makes the same map load within 4 seconds average... using async/await is averaging around 6 seconds

#

I tried the async/await approach to offload things from the main thread and reduce that time

#

I might be locking the thread at wrong points or might have to rework my file loading so it doesnt call apply to the textures right away

#

I'm not really experienced with neither unity nor threading so Im kinda lost

raw cargo
#

A common loading strategy with unity is to just live with it being non-smooth and somewhat slow… hide it behind a screen and carry on

slate wave
#

However I had to extract the bundle to the Resources folder