How do you setup data for one job using the results of another job

[polar beacon]

Hello, I have a set of jobs that I want to run sequentially. This of course is easy to do using the JobHandle from one as a dependency for another. However I need to use the results from one job to construct the data for the next job. So I am wondering what the best way to do this would be.

Originally I was going to do it in a 'setup' IJob that would run between the 'main' jobs and handle it, however I need to create Unity.Splines and that cannot be done within a job unfortunately as it requires IReadOnlyLists to setup properly.
And internally the NativeSpline creates an array and uses methods that are not compatible with Jobs.

Any advice would be great!

[hazy heart]

You combine the dependencies using JobHandle.CombineDependencies()

Thats easy if they are in the same system

Otherwise i think you'll need a singleton entity to store the info between the systems

[polar beacon]

I know, but as I said, I can't use a job to setup the data because the Splines API requires array[]

[hazy heart]

ok, then do that before on the main thread?

or use NativeArray then turn it into an array if the data for the array can be generated in the job?

[polar beacon]

I need to use the result from one job to create the splines that are then used in the second job

The end result being that I need to set the input for one job in the main thread that uses the output from another job.

[hazy heart]

then complete the job before its used on the main thread

[hasty vine]

normally you'd create your array with the tempjob allocator on the main thread before passing it into your job chain, the same array is passed into the first and second jobs, first would process and pass it on to the next.. the last job can use the [DeallocateOnJobCompletion] attribute to auto dispose the array when it's finished

[polar beacon]

I unfortunately don't know how many splines I will need before the job is completed which makes it hard to allocate the splines first.

[polar beacon]

I have job > job > job > setup data > job > job. So while it is possible, that would kind of be a pain to do and not really make sense imo.

[hasty vine]

not sure your exact setup obviously but can't you use nativelist?

[hazy heart]

Two ways to use native list:
if using ScheduleParallel native list needs a max capacity set outside the job.
Otherwise if you use Schedule it can grow in the job as required.

[hasty vine]

other option is create an entity on the fly for each spline, and add a dynamicbuffer of float3's or whatever you need to each, then use an IJobEntity to iterate/process all splines

[polar beacon]

Maybe, but really the issue is that NativeSpline cannot be created in a job, full stop.

[hasty vine]

Ah sorry, can't help you with that specifically i've never used it

[hazy heart]

are you sure?

[polar beacon]

Right, that is why I was asking how to setup data between jobs since it isn't really a Splines specific thing.

[hazy heart]

Remarks
NativeSpline is compatible with the job system

https://docs.unity3d.com/Packages/com.unity.splines@0.1/api/UnityEngine.Splines.NativeSpline.html

[polar beacon]

Yup, this is the constructor for it

public NativeSpline(IReadOnlyList<BezierKnot> knots, IReadOnlyList<int> splits, bool closed, float4x4 transform, Allocator allocator = Allocator.Temp)

Which uses this method in the CurveUtility class.

public static void CalculateCurveLengths(BezierCurve curve, DistanceToInterpolation[] lookupTable)

[hazy heart]

You should be able to make a TempJob NativeSpline and send it into a job

[polar beacon]

Sure you can send it to a job... from the main thread, like I said. I want to setup the data in the main thread after a job is done.

[hazy heart]

well then it seems like you'll just need to do the Complete()

[polar beacon]

When would I call it though? Not right away, that would defeat the main point of using jobs, wouldn't it?

[hazy heart]

doesnt defeat the point at all

[polar beacon]

It would block the main thread though wouldn't it?

[hazy heart]

if you do it for every system then its an issue

[hazy heart]

it forces the main thread to wait until all jobs have finished

[polar beacon]

Which would block the main thread no?

[hazy heart]

yes

but thats fine

it's just unavoidable in this case

[polar beacon]

Not for me it isn't if it can be helped at all.

[hazy heart]

i'd make a post in the splines forum (i assume there is one) about having a native array parameter, just in case they don't know

but for now you'll just need to complete

[polar beacon]

The best solution I have been able to think of is to store the handles in a list and have a component loop over them checking the isCompleted bool and once it is done, setup the data, and then schedule the jobs that depend on the splines.

But that will be a bit messy with my jobs design

[hazy heart]

honestly just profile it

you might find theres no difference

[hasty vine]

i think from what i can gather, i'd do it in two separate ISystems.. one that's unmanaged/bursted for all the setup jobs, and one that isn't bursted and pulls in the results from the first system to do the managed spline creation ( if i'm interpreting the problem correctly ).. i do agree that avoiding the mainthread would be better, i virtually never use Complete unless i'm backed into a corner by twelve snarling lions and don't have a machine-gun handy..

[polar beacon]

Hmm, not too sure how that would work if I am being honest.

Maybe the best solution is to just 'fork' the splines package and refactor it as needed to support creation within jobs. I guess that will be cleaner assuming all of the logic can be transferred over.

[hasty vine]

Well the first system could have a Singleton that holds the nativelists being added to and processed, then the second job could pull that singleton data in and use it to create the final splines.. again though, not sure of the exact setup, just a suggestion!

[thorny rampart]

Can't you just do a single Isystem and burst the separate jobs

[iron solstice]

@polar beacon Are you using Entities at all? I see some answers here assuming that you are, but I haven't seen you mention it.

[polar beacon]

I'm not. Just Jobs, which is probably why a few of the answers are a little confusing for me.

[iron solstice]

I'm wondering if you've tried to use the Spline API in a job, but without Burst? My understanding is that it's mainly Burst that limits what types you can use. The only limit is what types you can pass into a job, that's still always limited to blittables and native collections. But you should still be able to create managed arrays within the job and pretty much anything that isn't a Unity API that requires the main thread.

[polar beacon]

I am pretty sure I tried and got a error that only native collections could be created. But I don't remember now.
Either way, didn't realize that it was burst that was more limited. And not jobs as a whole. Might be worth giving another try.

[iron solstice]

I think the input and output is limited to the same types as Burst supports, whether you're using Burst or not. So even if you can create splines in the job, you might not be able to output it from the job, unless you encode it into a NativeArray somehow or bypass the safety system by setting a static variable.

[supple matrix]

Yeah, in general passing data from JobX to JobY requires you to pass it through a Native Allocation - Most of which can be found in the com.unity.collections package. So NativeArray or NativeReference or NativeList are examples. So long as both your jobs point to the same allocation (collection) then 1. you can freely pass data between jobs and 2. The Job System will kick in and error if one does not depend on the other.

Interesting point here is that you're allowed to allocate inside jobs. But, this rarely works great between jobs. Due to:

Each job also creates one Temp allocator per thread, and deallocates them in their entirety at the end of the job. Because a Temp allocator gets discarded as a whole, you don't need to manually deallocate Temp allocations, and doing so does nothing. (https://docs.unity3d.com/Packages/com.unity.collections@2.1/manual/allocation.html)
So instead, I suggest NativeList<T> it will allow you to at elements in one job iterate through em in another. Interestingly enough there are many Job types that support allocations in a pleathora of ways, to learn more, I suggest reading this: http://blog.s-schoener.com/2019-04-26-unity-job-zoo/ (by our very own Sebastian :3)

[polar beacon]

I appreciate the reply. The issue I have is actually because I need to create Unity.Splines.NativeSplines from the results of one job and use them in another job. But the NativeSplines cannot be created inside of a job as it requires using c# arrays, both as parameters in the constructor for it and within the body of the constructor.

[supple matrix]

Interesting, sounds like a fun problem - I'll see if I have time to look into it tomorrow, mostly because I too wanna create splines for my personal project :3

But i do wonder what your problem is? There's nothing stopping a job from containing managed data in a job. The only thing is that you just can't have a managed field in a job as it will null the value. Nothing is stopping you from getting it elsewhere. Like e.g a pointer or a static.