#Is readToEndAlloc still a good idea to use for loading entire file after Reader refactor

1 messages · Page 1 of 1 (latest)

candid cloud
#

I see that in the function readToEndAlloc is now marked as deprecated.
I am wondering if it is a bad idea to use it anyway if the intention really is to load the entire file into a single contiguous memory region.
To my knowledge introducing buffers through the new reader interface seems a little redundant if you can just load the entire file into memory with a single syscall (I assume readToEndAlloc gets the length of the file, allocates a space big enough for it and then does a sycall to load the file into the specified memory region).
Would really appreciate if someone more knowledgeable could tell me if this makes sense or correct any misconceptions.

#

You know what. I will just check myself. I can do the preformance measuring

lapis ridge
#

A streaming implementation of whatever needs the data in the file will usually be faster (as long as a reasonable streaming implementation is possible) than one that needs the file to be read into memory entirely but the win will usually be marginal.
Main issue of reading the whole file into memory is the memory usage if the file is large.
If memory usage and the marginal performance penalty arent a problem you can use std.Io.Reader.allocRemaining

candid cloud
#

I get that. And I agree. But when writing a code editor it makes sense

#

And I did the test. The Reader was about twice as fast as the readToEndAlloc on my machine.
Here is my test code if you are curious:

const std = @import("std");

pub fn main() !void {
    var debug_allocator = std.heap.DebugAllocator(.{}).init;
    defer _ = debug_allocator.deinit();
    const allocator = debug_allocator.allocator();

    {
        var elapsed_time: u64 = 0;
        for (0..100000) |_| {
            elapsed_time += try read_file_a(allocator);
        }
        std.debug.print("{d}\n", .{elapsed_time});
    }

    {
        var elapsed_time: u64 = 0;
        for (0..100000) |_| {
            elapsed_time += try read_file_b(allocator);
        }
        std.debug.print("{d}\n", .{elapsed_time});
    }
}

fn read_file_a(allocator: std.mem.Allocator) !u64 {
    const file = try std.fs.cwd().openFile("test.txt", .{});
    const start_time = std.time.Instant.now() catch @panic("Unimplemented");
    const contents = try file.readToEndAlloc(allocator, 0xFF_FF_FF_FF);
    const end_time = std.time.Instant.now() catch @panic("Unimplemented");
    allocator.free(contents);
    file.close();
    return end_time.since(start_time);
}

fn read_file_b(allocator: std.mem.Allocator) !u64 {
    const file = try std.fs.cwd().openFile("test.txt", .{});
    var read_buffer: [1024]u8 = undefined;
    var file_reader = file.reader(&read_buffer);
    const reader = &file_reader.interface;
    const start_time = std.time.Instant.now() catch @panic("Unimplemented");
    const contents = try reader.allocRemaining(allocator, std.io.Limit.limited(0xFF_FF_FF_FF));
    const end_time = std.time.Instant.now() catch @panic("Unimplemented");
    allocator.free(contents);
    file.close();
    return end_time.since(start_time);
}