How to read a wide string prefixed by its length from memory

[left sparrow]

So, I have a problem to solve where I have a pointer to the followng bytes in memory:
05 00 00 00 35 00 2E 00 34 00 2E 00 34 00 00 00
Its a struct, prefixed by its length as an i32 and then a null terminated widestring
here is the c++ struct definition, how can i port to zig and be able to read the string?
(because the length is explicitly given, the fact that the string is null terminated can be ignored)

struct __declspec(align(8)) System_String_Fields {
    int32_t _stringLength;
    uint16_t _firstChar;
};

like

const BoxedStr = extern struct {
  string_length: i32,
  first_char: ???
};

[ashen terrace]

The way you make flexible array members work is that you create a method to get the string.

I think there might be some good examples if you google flexible array member on the github issues.

[left sparrow]

Oh, good idea

[ashen terrace]

The old reddit has this which is how you do it:

https://www.reddit.com/r/Zig/comments/sr7p3f/does_zig_support_flexible_array_members/

[left sparrow]

pub const Il2CppString = extern struct {
    len: i32,
    data: [1]u16,

    pub fn getData(self: *Il2CppString) []const u8 {
        _ = self;
        // ...
    }
};

[ashen terrace]

Well if it's null terminated and the null termination doesn't include the length then you should return [:0]const u16

And if it's const then you should take const pointer to self

[left sparrow]

huh

oh i see

i dont think it should be null terminated actually

im not sure about that part

i think its just the length

it looks null terminated because of the padding bytes

i'm still strugglng

pub const Il2CppString = extern struct {
    len: i32,
    data: [1]u16,

    pub fn getData(s: *Il2CppString) []u8 {
        return std.unicode.utf16leToUtf8Alloc(
            std.heap.page_allocator,
            (&s.data)[0..@intCast(s.len)],
        ) catch [_]u8{0};
    }
};

and with this approach i would have to make sure to explicitly free the string returned from getData, wouldnt i

it works

[left sparrow]

is this true though? @ashen terrace

how would someone know they need to do this?

ig if i make the allocator a parameter that needed to be passed in all the way down

then it would be more obvious

pub fn getAppVersion(unity_player_base: usize, allocator: std.mem.Allocator) ?[]u8 {

there we go

fn resolveTypeInfos() void {
    const ally = std.heap.page_allocator;
    const game_version = general_utils.getAppVersion(
        module_base.unity_player,
        ally,
    ).?;
    defer ally.free(game_version);

    std.log.info("Game version is {s}", .{game_version});
}

Is this the proper, nice way to write this in zig?

i need to learn more abt allocator,s cause i don't even know if i should be using the page allocator for this

[ashen terrace]

Yea and also heap.page allocator is how you waste a lot of memory

Also your return catch thing is kinda broken. Just return the memory allocation error.

[left sparrow]

and then handle it higher up?

i switchted to c_allocator

[plush silo]

the convention in zig is to always take an allocator as a parameter if you need to allocate, and to never hard code one within a function, so resolveTypeInfos would also take an Allocator as a parameter

this convention is very helpful when e.g. testing, since you can swap out the allocator for std.testing.allocator and get leak checking/use-after-free checking/etc super easily

[left sparrow]

iteresting

[ashen terrace]

its not exactly a zig thing. Its a general programming principle called dependancy injection

basically if your thing A uses a thing B then rather than have thing A create a thing B make it so thing A is given a thing B.

That way you can use test versions of Thing B (like mock databases and things).

[left sparrow]

do you guys know how to do the opposite thing?

what if i want to convert a []const u8 to a widestr

[plush silo]

std.unicode.utf8ToUtf16LeWithNull or std.unicode.utf8ToUtf16Le or std.unicode.utf8ToUtf16LeStringLiteral, but you also need to be sure the []const u8 is UTF-8

[left sparrow]

I was a bit thrown off as to why utf8ToUtf16Le doesn't take an allocator?

but utf16LeToUtf8 does

[plush silo]

they're named strangely, usually utf8ToUtf16LeWithNull would be named utf8ToUtf16LeAllocZ, but it's common to have both an allocating version and a non-allocating version for things like this, where if you know your constraints (or can calculate them), then you can either allocate the necessary slice up front, or avoid allocation entirely and just use e.g. an array on the stack as the buffer

here's an example from the standard library:
https://github.com/ziglang/zig/blob/13c7aa5fefef9b1338951cf0d01c04345201d996/lib/std/child_process.zig#L1297-L1308

zig making allocation a concious choice like this is intentional as well, since if you can it's basically always better (i.e. faster) to avoid heap allocation

[left sparrow]

ok but now im very confused

i need to recreate the orignal struct

so theres the data: [1]u16 parameter i need to put there

with what i have allocated

@plush silo here is what i have:

pub const Il2CppString = extern struct {
    klass: ?*Il2CppClass_1,
    monitor: ?*anyopaque,
    len: i32,
    data: [1]u16,

    pub fn init(ally: std.mem.Allocator, s: []const u8) Il2CppString {
        const t = std.unicode.utf8ToUtf16LeWithNull(ally, s) catch null;
        return .{
            .klass = null,
            .monitor = null,
            .len = @intCast(s.len),
            .data = t,
        };
    }
//...

but i can't cast t into data field

and i have no idea where i should be freeing when i alloc here

the il2cppstring init call is into a variable on the stack

but has a field that will include data i need alloc to the heap. so im confused,, very

[visual talon]

t is optional

And your data field isn’t

[plush silo]

if Il2CppString expects data to actually be trailing and not a pointer, then you need to take a different approach. This is what I'd do:

pub fn init(ally: std.mem.Allocator, s: []const u8) !*Il2CppString {
    const utf16_len = try std.unicode.calcUtf16LeLen(s);
    const full_byte_len = @sizeOf(Il2CppString) + (utf16_len * 2);
    var string_bytes = try ally.alignedAlloc(u8, @alignOf(Il2CppString), full_byte_len);
    var string = @as(*Il2CppString, @ptrCast(string_bytes.ptr));
    string.* = .{
        .klass = null,
        .monitor = null,
        // double check if this expects length in UTF-16 code units or in bytes,
        // if it's bytes then do `utf16_len * 2` instead
        .len = utf16_len,
        .data = undefined,
    };
    var data_slice = @as([*]u16, @ptrCast(&string.data))[0..utf16_len];
    // catch unreachable since we already know `s` is valid UTF-8 from calcUtf16LeLen above
    std.unicode.utf8ToUtf16Le(data_slice, s) catch unreachable;
    return string;
}

pub fn deinit(self: *Il2CppString, ally: std.mem.Allocator) void {
    const byte_len = @sizeOf(Il2CppString) + self.len * 2;
    const byte_slice = @as([*]align(@alignOf(Il2CppString)) u8, @ptrCast(self))[0..byte_len];
    ally.free(byte_slice);
}

totally untested, so i might have gotten some of the casting and whatnot wrong but that'd be the idea--basically, allocate all the bytes necessary and then use the allocated bytes as the memory for the struct. you can't allocate a struct like this on the stack since data is variable length

here's another relevant example: https://ziggit.dev/t/objects-with-header-first-and-payload-after/1922/13

allocate @sizeOf(Header) + trailing_size bytes with alignment @alignOf(Header), cast it to *Header, and then make sure that you can still free the entire allocated slice (so either store the size of the trailing data in the header, or make it so the size of the trailing data can be calculated when it’s time to free).

[left sparrow]

wow, i appreciate the time you took to write this out for me. let me check out that article and read through what you wrote 🙂

[plush silo]

note also that the bytes of the [1]u16 itself are not really handled in the code i wrote, it's basically an extra 2 bytes that get allocated since it's included in @sizeOf(Il2CppString)

so if data should be null terminated then you should set data_slice[utf16_len] = 0; before returning from init (this is usually what the [1]u16 field would be useful for), or if it doesn't need to be null-terminated then you can either allocate 2 fewer bytes or change the definition of Il2CppString to something that doesn't include the [1]u16 in some way (not sure what the best way to do that would be beyond just removing the field entirely)

EDIT: data: [0]u16 appears to be legal, so that might work