#json: Parse into an ArrayList directly

1 messages · Page 1 of 1 (latest)

bronze frost
#

I have a really complex type, and I need it to be dynamically allocated...
but I also need to be able to parse json data into it without having to create my own parser for the entire type.

Is it possible to not have to reinvent the wheel here?
Is there a way to make the std.json parser understand that a JsonArray needs to go into an ArrayList?

topaz wing
bronze frost
topaz wing
# 
const std = @import("std");

const MyStruct = struct {
    a: u32,
    b: std.ArrayListUnmanaged(u32),

    pub fn jsonParse(
        allocator: std.mem.Allocator,
        source: anytype,
        options: std.json.ParseOptions,
    ) !MyStruct {
        const TmpType = struct {
            a: u32,
            b: []u32,
        };

        const parsed = try std.json.innerParse(TmpType, allocator, source, options);
        return .{
            .a = parsed.a,
            .b = .fromOwnedSlice(parsed.b),
        };
    }
};

pub fn main() !void {
    var debug_allocator: std.heap.DebugAllocator(.{}) = .init;
    const allocator = debug_allocator.allocator();

    const x = try std.json.parseFromSlice(MyStruct, allocator, "{\"a\": 123, \"b\": [1, 2, 3]}", .{});
    defer x.deinit();

    for (x.value.b.items) |y| {
        std.log.debug("{}", .{y});
    }
}
#

Something like this. Note that this code is very bad and you definitely shouldn't use it. If you insert something to b, you'll get double free most likely.

bronze frost
topaz wing
#

I think the easiest way is to create a second type with slices instead of ArrayList, then copy it to your original type

bronze frost
#

Is there really no way around this? It sounds really weird that there isn't

topaz wing
#

It sounds really weird that there isn't
I think that's because parsing functions use a temporary arena allocator, so you can easily call .deinit() to free the whole struct

bronze frost
#

Sure, but you did fromOwnedSlice, which removes the ownership from the arena
How does that translate into a double free or to being bad code?

topaz wing
#

Yes, but the arena allocator doesn't know about it

#

and it will try to free it anyway

bronze frost
#

Doesn't fromOwnedSlice create a copy?
what's the point of that function if that's not the case? 🧩

topaz wing
#

Doesn't fromOwnedSlice create a copy?
no

#

You could create a copy yourself, but there's an issue with that

#

You don't know which allocator to use

bronze frost
#

or is that not accessible from jsonParse?

topaz wing
#

it's not accessible

bronze frost
#

ouch

topaz wing
#

maybe I have an idea

#

1s

# 
    pub fn jsonParse(allocator: std.mem.Allocator, ...) !MyStruct {
        const arena: *std.heap.ArenaAllocator = @ptrCast(@alignCast(allocator.ptr));
        const child_allocator = arena.child_allocator;
    }
#

I think you can do something like this to get the allocator you passed to std.json.parseFromSlice

#

but it's kinda ugly

#

and you still have to do a lot of work yourself

bronze frost
#

Its generalizable, though. so the code generator can copy/paste the function into all types it creates 🤔

#

why is that ugly/bad code still?

topaz wing
#

so the code generator can copy/paste the function into all types it creates
I'm not sure I understand

bronze frost
#

My type is created from a json schema definition
I'm creating the zig type declarations from that schema

topaz wing
#

why is that ugly/bad code still?
we make assumptions about inner workings of the json parser

topaz wing
bronze frost
topaz wing
bronze frost
topaz wing
bronze frost
#

Wait, is child_allocator the backing allocator of the arena, that you pass into the arena itself on init?

topaz wing
#

yes

bronze frost
#

oh, yea that's not good

# 
const std = @import("std");
const ArrayList = std.ArrayList;

const MyStruct = struct {
    items: ArrayList(i32),

    pub fn jsonParse(
        allocator: std.mem.Allocator,
        source: anytype,
        options: std.json.ParseOptions,
    ) !MyStruct {
        const json_value = try std.json.innerParse(std.json.Value, allocator, source, options);
        return try jsonParseFromValue(allocator, json_value, options);
    }

    pub fn jsonParseFromValue(
        allocator: std.mem.Allocator,
        source: std.json.Value,
        options: std.json.ParseOptions,
    ) !MyStruct {
        _ = options;
        const arr = source.object.get("items").?.array;
        
        var items = ArrayList(i32).init(allocator);
        for (arr.items) |item| {
            try items.append(@intCast(item.integer));
        }
        
        return .{ .items = items };
    }
};
``` @topaz wing any thoughts on this? 👆
I don't know if its any different than you alternative without `jsonParseFromValue` 🤔
topaz wing
#

hmm

#

if you look like at the source code of parseFromValue

#

maybe you could use parseFromValueLeaky, so you can your actual allocator, and not the arena

#

anyway, I have to go now

#

good luck with your project

bronze frost
# 
const MyStruct = struct {
    const T = u32;
    a: T,
    b: std.ArrayListUnmanaged(T),

    pub fn jsonParse(
        allocator: std.mem.Allocator,
        source: anytype,
        options: std.json.ParseOptions,
    ) !MyStruct {
        const parsed_a = try std.json.innerParse(T, allocator, source, options);
        const parsed_b = try std.json.innerParse([]T, allocator, source, options);

        const data_a = try std.json.parseFromValueLeaky(T, allocator, parsed_a.value, options);
        const data_b = try std.json.parseFromValueLeaky([]T, allocator, parsed_b.value, options);
        return .{
            .a = data_a,
            .b = .fromOwnedSlice(data_b),
        };
    }
};
```I can't figure this out 🙁
#

parsed_a and parsed_b do not even have a .value field!

hidden remnant
#

Are you using the Json arraylist?

#

or the normal arraylist

bronze frost
#

Never heard of that

hidden remnant
#

its a wrapper that allows it to be parsed

#

i dont actually recall the specifics

#

but i had to use it recently

bronze frost
hidden remnant
# 

files: std.json.ArrayHashMap(FileData.Hash) = .{ .map = .empty },
comment: ?[]const u8 = null,


pub fn toWriter(self: *const ProjectVersion, writer: *std.Io.Writer) !void {
    var formatter = std.json.fmt(self, .{});
    try formatter.format(writer);
    try writer.flush();
}

pub fn fromReader(allocator: std.mem.Allocator, reader: *std.Io.Reader) FromReaderError!ProjectVersion {
    var allocating_writer = std.Io.Writer.Allocating.init(allocator);
    defer allocating_writer.deinit();

    _ = try reader.streamRemaining(&allocating_writer.writer);
    try allocating_writer.writer.flush();

    const json_data = try allocating_writer.toOwnedSlice();
    defer allocator.free(json_data);

    var parsed: std.json.Parsed(ProjectVersion) = try std.json.parseFromSlice(ProjectVersion, allocator, json_data, .{});
    parsed.deinit();

    return parsed.value;
}

This is my old code i managed to scrape together again

#

I havent copied the whole thing cuz it's much too large

#

but basically do as in the sample

#

and "wrap" your array type

bronze frost
bronze frost
hidden remnant
#

Ah i see

#

mb