How to guarantee struct field ordering with ArrayList?

[formal tide]

I am using shared memory to transfer a large amount of data between processes. This data is mostly formed of an array of structs, to do this, these structs needs to have consistent ordering across processes (e.g. extern struct). In trying to do this I encounter the error: error: extern structs cannot contain fields of type 'array_list.ArrayListAligned(u8,null)'
E.g.:

const MyType = extern struct {
    const Self = @This();

    fd: i32,
    data: std.ArrayList(u8),

    fn init(allocator: Allocator, fd: i32) !Self {
        const arr = try std.ArrayList(u8).init(allocator);
        return Self {
            .fd = fd,
            .data = arr
        }
    }
}
fn main() !void {
    var buffer: [100_000]u8 = undefined; // This would be the shared memory
    var fba = std.heap.FixedBufferAllocator.init(&buffer);
    const allocator = fba.allocator();

    var snapshots = std.ArrayList(MyType).init(allocator);
    snapshots.append(MyType.init(allocator,21));
}

I could implement my own version of ArrayList here to solve this, but I really don't want to do this. Is there a way to apply extern to external structs?

How to guarantee struct field ordering with ArrayList?

[obsidian carbon]

std.ArrayList() returns a normal zig struct, normal zig structs have no defined layout

• they also contain a slice, which has no defined layout

[formal tide]

Okay.

This can be done, for example in Rust:

#[repr(C)]
struct MyStruct {
  fd: i32,
  data: Vec<u8>
}

[formal tide]

This restriction also seems to apply to the allocator

If I have to rewrite both these structure for this trivial change, this seems like quite an issue

[obsidian carbon]

also, you don’t really need to store they whole ArrayList, you only need to store the items field, which you can construct an arraylist out of on the other side

[formal tide]

It seems []T does also not allow extern

It seems quite a large problem you can't use most of the standard library if you need this

[obsidian carbon]

most of the standard library is for zig, not for passing around to extern functions

[obsidian carbon]

the "equivalent" of this would be

extern struct {
  fd: i32,
  data: [*]const u8,
  len: usize,
  capacity: usize,
}

which you can package before sending to whatever needs a fixed format, and unpackage on the other end

normal structs and slices explicit avoid having a defined layout so that the compiler can optimize them as needs be

[formal tide]

Okay, how would I use [*]const u8 and usize withallocator.alloc() and allocator.free()?

[obsidian carbon]

to go from struct to slice

var slice: []const u8 = undefined;
slice.ptr = thing.data;
slice.len = thing.len;

to go from slice to struct

thing.data = slice.ptr;
thing.len = slice.len;

[formal tide]

Would I also need to re-implement std.heap.FixedBufferAllocatorwith the extern tag since I want to store this in the shared memory as well?

[obsidian carbon]

well, firstly none of the allocators (besides std.heap.c_allocator) are threadsafe, so you need to really consider who owns what memory

[formal tide]

I am aware, the memory is not accessed simulatenously. It is simply used to transfer a large amount of data between processes. This is more efficient than using sockets for my case.

[obsidian carbon]

between processes?

[formal tide]

Yep

[obsidian carbon]

you can't just share an allocator across processes

they have separate address spaces

[formal tide]

Linux can handle pointer adjustments when attaching shared memory

[obsidian carbon]

you can certainly reconstruct a FixedBufferAllocator on the other side, but you can't just throw the allocator to the other side

[formal tide]

It is relatively easy to implement an allocator you can throw between processes

It stores addresses as offsets from the attached shared memory address

[obsidian carbon]

sure, but none of the allocators in the stdlib do that, you're venturing into territory that the stdlib hasn't really every focused on

[formal tide]

That's fair

In a way I'm implementing a server that can update the process without dropping client streams

It's a little specific

Do you know the language policy on allocators? If I made a PR for a shared memory allocator would this be considered? (if this is even possible in the context)

[smoky citrus]

that's a pretty verbose version of const slice = thing.data[0..thing.len];

[obsidian carbon]

fair, both work

[smoky citrus]

as it stands, all of the public allocators are cross-platform abstractions (at least for tier 1 supported targets), so I think that would be the bar.

though beyond that, std.mem.Allocator isn't really meant/designed for ABI or inter-process work - as stated by RIP, it's mainly meant as an "inside of zig" thing

[humble cloak]

While inside of zig is definitely nice and cozy I do I find that a lot of the times the need for extern comes. So @formal tide point is also correct and I understand it.

[obsidian carbon]

thats why extern exists, but that doesn't mean that every single stdlib structure should be extern just because you want it to always have an explicit layout sometimes

[humble cloak]

i never said anything should be extern i dont know why you're saying this

i dont need any extern functionality out of the box, it's just about making it possible in a totally not dumb way

some std stuff is easy to extern

[obsidian carbon]

the point of this post was having std.ArrayList be extern, and then making an allocator extern

[humble cloak]

the functionality of it

of course you have to do it by hand

but how?

[obsidian carbon]

the same way you make any other struct, you define fields, except use extern struct instead of struct

[humble cloak]

functionality

[obsidian carbon]

if you wanted it to work identically, the code is in the stdlib for you to view