#read from file to array list with new Io.Reader api

1 messages · Page 1 of 1 (latest)

tardy spire
#

I can't tell whats wrong with this code. Its hanging at the second line from bottom. I would expect it to read the entire file into the array list.

const std = @import("std");

test {
    var dir = std.testing.tmpDir(.{});
    defer dir.cleanup();
    var f = try dir.dir.createFile("foo", .{ .read = true }); // create file for writing and reading
    defer f.close();
    try f.writeAll("foobar" ** 10);
    try f.seekTo(0);
    var buf: [std.heap.pageSize()]u8 = undefined;
    var freader = f.reader(&buf);
    var al = std.ArrayListUnmanaged(u8){};
    var w = std.io.Writer.Allocating.fromArrayList(std.testing.allocator, &al);
    _ = try freader.interface.streamRemaining(&w.writer); // hangs here
    std.debug.print("{s}\n", .{al.items});
}

I'm using this zig version which i think is the most recent tarball available.

$ zig version
0.15.0-dev.1034+bd97b6618

Also, is this the correct way to init a File.Reader? I assume you can provide any sized buffer you wish but thats just a guess.

tropic wedge
#

there's a couple issues with this code, but the basic premise is correct

#

it shouldn't be hanging

#

let me do some testing

#

okay it's a bug in std.fs.File.Writer.sendFile, it seems to be returning 0 when it should return error.EndOfStraem

tardy spire
#

ah i wondered if it might be something like that. i was only looking in streamRemaining() though.

tropic wedge
#

i'm not 100% sure what it should be doing, but the implementation does return 0 all over the place which is almost certainly wrong

#

this might be left over from a previous version of writergate

tardy spire
#

you think i should post this on zulip#writergate or open an issue on github?

tropic wedge
#

i'd just fix it and send a PR, personally :)

#

shouldn't be too hard

#

but a zulip post might not be a bad idea either

tardy spire
#

maybe i'll take a look. i'm a little worried if you're confused by the code though. maybe i'll take a look and if its not clear to me, will post on zulip.

#

thank you for the advice!

#

🙏

tropic wedge
# tardy spire I can't tell whats wrong with this code. Its hanging at the second line from bo...

here's a couple other issues with the code you've written, for after this gets fixed:

  • you don't need to use Allocating.fromArrayList here, just init it
  • if you do use fromArrayList, be aware that it becomes the owner of the arraylist's memory - the original arraylist is cleared, and will not be updated with the data written to the Writer.Allocating
  • if you want an arraylist from a Writer.Allocating, you need to use toArrayList
  • if you know you're only using the file with a single reader, it's probably slightly better to use readerStreaming rather than reader, since that gets the kernel to update the file offset rather than having to use pread every time - i'm not sure if this actually makes any difference though, and reader is safer because you can create multiple independent readers from a single file
#

oh, and for initializing arraylists, use var al: std.ArrayListUnmanaged(u8) = .empty

#

decl literals are good, and default initializers are bad (in this case)

#

you can also use decl literals for Writer.Allocating: var w: std.io.Writer.Allocating = .init(std.testing.allocator);

tardy spire
#

thanks. yeah this isn't what i want. in my actual code i'm trying to re-use the array list's allocated memory if possible. do you happen to have any ideas for that?

#

in the test, of course there is no memory yet.

#

i'm just hearing about the different reader modes so glad to learn about readerStreaming

#

i guess maybe its not a great idea to try to re-use the array list's allocation. might have to eliminate that use case from my parser.

#

...since it doesn't seem like a supported use case in io.Writer.Allocating. i think instead of accepting *Reader, i should just accept a File and do 1 allocatation every time instead of trying to reuse the memory.

tropic wedge
#

if you just want to retain the backing buffer, and don't actually need an ArrayList specifically, you can just use Allocating.clearRetainingCapacity()

tropic wedge
#

unlike the old pattern of ArrayList.writer(), you have to explicitly pass the ownership of the buffer back and forth between the ArrayList and the Writer.Allocating

#

(and it definitely is a supported usecase ^-^)

tardy spire
tardy spire
#

@tropic wedge i found a way to reuse array list memory with Reader.appendRemaining(). just wanted to run it by you. not sure about the code i posted in that issue above, but this is working nicely in this test and also in my simdjzon parser. please forgive the sloppy code.

const std = @import("std");

test {
    var dir = std.testing.tmpDir(.{});
    defer dir.cleanup();
    var f = try dir.dir.createFile("foo", .{ .read = true }); // allow reading and writing
    defer f.close();
    const s = "foobar" ** 10;
    try f.writeAll(s);
    try f.seekTo(0);
    var freader = f.reader(&.{});
    var al: std.ArrayListUnmanaged(u8) = .empty;
    defer al.deinit(std.testing.allocator);
    _ = try freader.interface.appendRemaining(std.testing.allocator, null, &al, .unlimited);
    try std.testing.expectEqualStrings(s, al.items);
}
tropic wedge
tardy spire
#

nothing really except i couldn't seem to figure out how to make it work. i did try using from/toArrayList but it was hanging like the code in that issue.

#

this just works

tropic wedge
#

oh, right yes the sendfile bug I remember now

tropic wedge
#

though actually for this usecase it makes more sense to copypaste the contents of that function and tweak it a bit

tardy spire
#

cool! i have a lot to learn about the new I/O api.

#

yeah i think appendRemaining() is just what i want since it doesn't take ownership of the array list.

tropic wedge
#

sure :)

#

have a look at Reader.expandTotalCapacity and Reader.fillMore too, just to get familiar with all the options :)

#

expanding the reader's internal buffer may be more efficient? i'm not entirely sure

#

and you can obviously move the buffer out of the reader and into an arraylist when you're done reading