#Choosing the right persistent, job-friendly asset workflow

1 messages · Page 1 of 1 (latest)

sonic bobcat
#

I’ve spent some time researching how to set up a persistent, job-friendly asset pipeline in Unity, but I’m still not sure which path makes the most sense.

Here are my requirements:

  • The asset must be persistent (saved in the project, version-controlled).
  • I need to build authoring/editor tools so users can edit/import data offline in the Editor.
  • My project uses GameObject/MonoBehaviour, not ECS.
  • The asset should be readonly at runtime but referenceable by MonoBehaviours.
  • Jobs must be able to use the asset data (i.e. data must be in native memory, Burst-safe).
  • Multiple MonoBehaviours may reference the same asset in parallel.
  • Asset size could be large (4–100 MB).

Options I’m considering

  1. Naive ScriptableObject
    Supports custom editors, inspector integration, easy to reference.
    Not job-friendly: each MonoBehaviour would need to copy into NativeArrays on Awake, leading to duplicated memory.

  2. BlobAssets (without full ECS)
    Idea: use a ScriptableObject for metadata & tooling, and hold a BlobAssetReference inside it.
    Workflow: users import data → adjust → click “Bake” → baked asset is stored.
    Problems:
    BlobAssetReference isn’t serializable, so I can’t persist the baked blob in the SO.
    If I build it at runtime, I lose editor access and have to manage disposal carefully.
    If I follow the ECS authoring flow, I end up pulling in Entities just for blobs, which feels heavy.
    Net result: fiddly lifetime management and dependency creep.

  3. ScriptedImporter
    Haven’t dug deep yet, but seems like I could define a custom .asset or .bytes file type and bake directly into that format.
    Might solve the persistence and readonly requirements cleanly.

  4. Addressables / RawFiles
    Could bake out a binary blob as a RawFile and load it directly into a NativeArray<byte> via AsyncReadManager.
    Good for large assets (streamable, no managed copy).
    More build-pipeline setup, probably overkill if assets aren’t that big.

Which of these 4, or else, are my best option going forward?

livid moss
sonic bobcat
#

I need to store a graph with thousands of nodes and edges with unmanaged types stored at each node.

livid moss
#

does that data need to be imported at runtime?

sonic bobcat
#

Imported offline in the editor but referenceable at runtime by several GameObjects.

livid moss
#

do these GO references need to be editable in the editor?

#

Or can they be ID lookups or procedures?

sonic bobcat
#

Yeah there would be a serialisable reference on this component that would need to point to the asset.

#

The graph asset.

livid moss
#

So no deep references into the graph?

sonic bobcat
#

As in to specific nodes?

#

As a temporary fix I was using ScriptableObjects with CSR adjacency arrays and then copying them into NativeArrays for each GameObject so they can use the graph in jobs.

livid moss
#

do you want to edit the graph nodes and edges in the unity editor directly or is the graph always baked as a whole?

sonic bobcat
#

The entire graph is baked procedurally.

#

And then readonly after that.

#

So if you wanted to change the graph you would have to change the parameterisations and then rebake.

#

In what ever authoring tool I end up using.

livid moss
#

then you store the graph as a binary blob of plain & minimal DTO c# types in a SO field and on load construct your optimized collections

sonic bobcat
#

Alright I'll have a look. Thanks!

#

So you dont think I can avoid copying the SO field for each GameObject?

livid moss
#

the performance magic happens when your blob is chunked such that you can use it in small bits on demand (streaming)

sonic bobcat
#

ahhh

#

ok

livid moss
#

The runtime graph should be held in one location in memory.

#

typically the SO would be owned by a unique system/manager and the GOs get their data by talking to that manager

sonic bobcat
#

Right so on the SO i have an editor tool and the user clicks bake. Then theres like a byte array stored on some static class that manages all the graph instances and I can pass that data safely to the runtime jobs and they can access it via the static instance?

livid moss
#

If you decide to use native Unity serialization on the SO to load/save your graph, you don’t need the manager ofc.

livid moss
sonic bobcat
#

I cant just use a byte[] correct? This is considered managed by jobs?

livid moss
#

also, that byte array is only the storage format. It might not be necessary to go that far.

livid moss
#

And they are managed, not usable in jobs

sonic bobcat
#

So I create some value type representation of a graph. Store references on a static class so they are globally accessible. And the jobs get the pointer to from the SO.

#

Something like that.

livid moss
#

Jobs don’t use pointers

#

they use copies of data or slices of copies

#

it might be best to get the data persistence sorted separately from the job/runtime usage of that data. In your situation I’d aim to not optimize both things at the same time. Expect a conversion step to transform persistence data to runtime data.

#

to make loading efficient, you need to know exactly what the data access patterns will be, and until the jobs are well defined and optimized that isn’t the case.

sonic bobcat
#

Im running pathfinding over graph.

#

Let me get an example.

livid moss
#

figured as much

#

You could get the free version of a*pathfinding project and look at the persistence and serialization there to get a complete example

sonic bobcat
livid moss
#

It’s not the greatest way to do it

sonic bobcat
#

So earlier today I was doing something similar to what I think you were saying before.

#

Where i just stored my graph representation using normal methods like int[] edges and what not.

livid moss
#

and that wasn’t good enough?

sonic bobcat
#

And in the awake of my GO components I would create this and dispose

image.png
#

I was just concerned that since the graph could get quite large it would be a bad idea to copy the graph representation into all these arrays for each game object reading it

#

Unless I am misunderstanding how NativeArrays work

#

I have jobs that need to read the whole graph during their runtime meaning I cant temporarily load in slices of the complete graph. And these jobs are also invoked ~ 20 - 60 times per second

#

This was one of my jobs while I was still trying to use BlobAssets

image.png
#

Anyways, I'll try piece it all together. This was helpful, thank you.