#Is there an equivalent to C# List for the Job system

1 messages · Page 1 of 1 (latest)

little mantle
#

Hello!
I'm trying to make a job that would create big meshes. I don't know how many vertices it will have to create and not even how many meshes (without the job system, I'm creating a List<List<Vector3>> for the vertices for example, and I fill a list until it reaches 65000 vertices, then add a new one, ...).

But I couldn't find how to do that with the job system since when declaring a NativeArray, I have to give it its size.
The only way I found to add an element is to create a new NativeArray with a bigger size and then copying the content of the old one to the new one with a for loop but I guess it's not great for performance...

pallid glen
little mantle
#

Wow thank you! There are a lot of them actually 😂
Is there one that would best fit what I need? I don't know which one to choose 😅

pallid glen
#

you'd need to rethink your design potentially

#

because List of Lists is not supported by Job's safety system

little mantle
#

Oh ok

#

That's annoying because it was actually a list of lists of lists since I was trying to create a IJobParallelFor, each job creating a few meshes 🥲

#

What about a NativeArray of "Lists" (whatever equivalent) of NativeArrays?
Since I know how many jobs I will create (so I can use NativeArrays for the 1st level) and I can create NativeArrays of 65000 elements for each mesh (so I can also use NativeArrays for the 3rd level)

pallid glen
#

so anything not-flat will result in throws and will require you to disable safety for job

little mantle
#

😭

#

I don't get it, how is it possible to do anything without ever using nested Native things?

pallid glen
#

it's possible to do on main thread 😅

little mantle
#

Ok... 😅
I guess I'll use normal C# threads, dod is too hard for me 😫

acoustic lotus
#

You can use NativeList of UnsafeLists

#

i.e. NativeList<UnsafeList<float>>

acoustic lotus
#

And this is, like Issue says, is safety disabling.

little mantle
#

Alright I'll look at that, thank you!
But I think I first need to learn a lot more about Unity DOTS before doing complicated things like that 🥲

waxen spade
#

why not just create multiple jobs, one list each?

little mantle
#

And even if I did, from what I understand, a IJobParallelFor can't output an array since nested arrays are not allowed (and an array is already required to store the results of the jobs).
Am I understanding something wrong?

pallid glen
#

and use second list to figure first index of vertex in arrray per mesh

#

so you'll have 65k * numOfMesh length of NativeList

little mantle
#

But how can one job know how much meshes the "previous" jobs created? (And therefore the index to start setting its vertices)

pallid glen
#

that's what second list is for

#

it'll contain indices of first list

#

indices that point to first vertex of mesh

#

so assumming you have 3 65k meshes

little mantle
#

But maybe a job will finish before it the ones before in the 1st list (so it wouldn't know its index yet)

pallid glen
#

you'll have second list of { 0, 65k, 65k*2 }

pallid glen
#

I'm guessing

#

the algo to determine length of vertices array is separate from processing those vertices

#

is that right?

little mantle
pallid glen
#

well, in that case job might not be parallel

#

but it's still multithreading

little mantle
#

But actually, I could have a third list that would contain the orders in which the jobs ended right?
This way a job sets its vertices in the 1st list whenever it finishes and I can then reorder the list in the main thread 🙂

pallid glen
#

you only need one job here

#

(from what I see)

little mantle
#

Isn't it one job for every index of the parallel for?

pallid glen
#

no way

little mantle
#

Oh but then what's the point?

pallid glen
#

you don't run it on main thread?

#

you run it bursted

little mantle
#

But then what's the difference between a parallel for job and a normal job with a normal for loop?

pallid glen
#

parallel jobs split batches between worker threads

#

but since your algo doesn't know the length of data beforehand

#

you can't really do it

#

because to run parallel job you need to define the lenght of array you process

little mantle
#

Oh yes true

#

Oh so you mean I should do a normal for loop in a normal job to still benefit from burst?

pallid glen
#

it's also going to be async

#

so you can schedule at start of frame

#

and force complete at end

#

or don't force compelte at all

little mantle
#

But I would still need the length of the Array before

pallid glen
#

and just wait between frames

pallid glen
#

in a way

#

so it's resized easily

#

in parallel you can't resize anything

little mantle
#

NativeList<NativeArray<float3>> or NativeList<float3> and then I split it in seperate meshes in the main thread?

pallid glen
#

the latter

#

basically

#

NativeList<float3>
NativeList<ind>

#

first is flat array for indices of all meshes

#

second is index array which says at which index which mesh starts

little mantle
#

Alright thank you very much it's much clearer now! 😁

#

And so if I tried resizing a NativeList in a parallel for, it would throw an error?

pallid glen
#

which does not support resizes

#

in short: it's not safe and if you resize on one thread, others might keep writing to previous pointer

#

which leads to data loss + potential crash

little mantle
#

Ok thank you! 🙂

waxen spade
#

are you on a platform where you can't use 32bit indicies?

pallid glen
#

seems like 16 bit is a case for old android

little mantle
#

Oh I didn't know that, thank you! 🙂

little mantle
#

I finished rewriting the code with DOTS, it works perfectly (and it's super fast 😮)! 😁
Thanks a lot for your advices and ideas! 🙂

rigid onyx
little mantle
#

(I'm still using 16-bit meshes for now though to make a chunk system and not load the whole terrain with 1 mesh otherwise it takes way too much memory...)

cedar zealot
#

can burst vectorize unsafelist iterations too?

pallid glen
pallid glen
#

burst is going to optimize it, but I think your best shot is vectorization of looped list inside loop

#

something like:
foreach (var list in lists){
vectorizedforeach(var elem in list)
}

cedar zealot
#

I'm just asking if a plain UnsafeList can be vectorized

pallid glen
#

sure, why not

cedar zealot
#

is that how I create custom container types?

#

like a grid

pallid glen
#

NativeList is pointer to UnsafeList

#

UnsafeList can be easily reinterpreted to NativeArray

cedar zealot
#

can I give my custom container safety features somehow?

#

like, the underlying one wont have

pallid glen
#

yeah, but I'm not aware how

#

check out collections manual

cedar zealot
#

ah okay

pallid glen
#

or try to follow other implementation of native containers

#

tertle's core has many for sure

#

NZcore from Enzi also has some

cedar zealot
#

oh thank you!

#

I wish there were official documentation or tutorials on how to create something like this

#

seems really useful

pallid glen
#

collections manual might have one

#

all of those native containers thingies are from it

cedar zealot
#

oh awesome, I'll look into it!

#

thanks!

acoustic lotus
cedar zealot
#

oh just what I needed! Thanks!

rigid onyx
#

while I maintain the only time to worry about vectorization is when you're actually checking the assembly output for whether it's vectorized, iirc native containers are flagged by burst as non-aliasing, whereas unsafe containers aren't, which can help improve the chances of vectorization for native containers in some cases. so they aren't necessarily the same, I don't think.