Read/Write Blob Assets from Disk

[stiff owl]

Is there a way to Read or Write Blob Assets from disk?

[stiff owl]

I found the documentation for BlobAssetReference.Write and BlobAssetReference.TryRead

Took awhile to find it with Google, probably should have looked directly.

[queen needle]

I am successfully using BlobAssetSerializeExtensions.Write and BlobAssetSerializeExtensions.Read for my blob caching.

[median yew]

Same here. These are extension methods for classes that implement the interface BinaryWriter and BinaryReader. For whatever mysterious reasons, MemoryBinaryWriter is implemented public, and StreamBinaryWriter internal. So I ended up copying the latter in my own public class. (slightly modified to do compression)

[stiff owl]

compression sounds nice

does it add much time to your reading/writing to do that?

[pastel iron]

I have a blob asset that's generated from a csv file, only do it once at startup so time doesn't really matter and I haven't measured it, but I know it's fairly quick

[median yew]

Cannot recall the profiling result, just generally that adding compression on the memory stream was insignificant compared to the order of magnitudes slower serialization to disc (/NVMe). Which is why I implemented that asynchronous as done by DOTS serialization using AsyncReadManager

[stiff owl]

Still so much stuff to learn

So I can read a blob asset asynchronously with this?

[median yew]

DOTS serialization (used by Entities package to serialize and deserialize subscenes) contains everything you need to know how to do this. I found it not easy to understand though and spent too much time on this. The code I use to start deserializing a blob async looks rather innocuous: var fileInfo = AsyncReadManagerHelper.GetFileInfo(FileHelper.noaaProductCatalogueBlobPath); if (fileInfo.FileState == FileState.Exists) { //Debug.Log($"Start reading {result.FileState}"); unsafeFileHelper = new UnsafeFileHelper(FileHelper.noaaProductCatalogueBlobPath); readHandle = unsafeFileHelper.ScheduleFileRead(fileInfo); }

And finalizing it whenever it is ready a couple of frames laterif (readHandle.Status == ReadStatus.Complete) { noaaProductCatalogueReference = unsafeFileHelper.LoadBlobFile<NOAAProductCatalogueBlob>(); readHandle.Dispose(); unsafeFileHelper.Dispose(); AsyncReadManager.CloseCachedFileAsync(FileHelper.noaaProductCatalogueBlobPath); }

The UnsafeFileHelper struct looks a bit hairy though. All inspired by DOTS serialization.

[median yew]

Here the version without compression: public struct UnsafeFileHelper : IDisposable { bool memoryAllocated; NativeArray<ReadCommand> m_cmds; FixedString512Bytes m_path; public UnsafeFileHelper(string path) { m_cmds = new NativeArray<ReadCommand>(1, Allocator.Persistent); m_path = path; memoryAllocated = false; } public unsafe BlobAssetReference<T> LoadBlobFile<T>() where T : unmanaged { using (var memoryBinaryReader = new MemoryBinaryReader((byte*)m_cmds[0].Buffer, m_cmds[0].Size)) { var featureCacheBlobReference = memoryBinaryReader.Read<T>(); return featureCacheBlobReference; } } public unsafe ReadHandle ScheduleFileRead(FileInfoResult fileInfo) { memoryAllocated = true; ReadHandle readHandle; ReadCommand cmd; cmd.Offset = 0; cmd.Size = fileInfo.FileSize; cmd.Buffer = (byte*)UnsafeUtility.Malloc(cmd.Size, 16, Allocator.Persistent); m_cmds[0] = cmd; readHandle = AsyncReadManager.Read(m_path.ToString(), (ReadCommand*)m_cmds.GetUnsafePtr(), 1); return readHandle; } public unsafe void Dispose() { if (memoryAllocated) UnsafeUtility.Free(m_cmds[0].Buffer, Allocator.Persistent); m_cmds.Dispose(); } }

[stiff owl]

truly the lowest level coding I have ever seen in C#