#Why max_size is required for reading a file ?

1 messages · Page 1 of 1 (latest)

pseudo totem
#

There are many ways to read a file with the help of Reader and store data into a buffer/slice i.e, []u8 and each require some how to define the max size.

Fixed Buffer (inherently needs size):
One way directly create an empty array and send it as a slice for filling it (Here we need to define the size of array) fn read(self: Self, buffer: []u8)

Array List and max_append_size:
The other way is to initialize Arraylist but still the method needs to know the max size fn readAllArrayList( self: Self, array_list: *std.ArrayList(u8), max_append_size: usize)

Allocator and max_append_size:
The other way is to send an allocator that will create an arraylist behind and store the data fn readAllAlloc(self: Self, allocator: Allocator, max_size: usize,)

Delimiter :
This method: fn streamUntilDelimiter(self: Self, writer: anytype, delimiter: u8, optional_max_size: ?usize,)seems to be promising but needs delimiter, that doesn't make sense for a file.

#

Even if max size is kept to MAX of usize it's capped to 4KB internally in internal function call: try array_list.ensureTotalCapacity(@min(max_append_size, 4096));

young plaza
#

it's not capped at 4kb, it just preallocates that much

#

if it needs more then it expands later in the function

#

presumably so it doesn't try to allocate 19 exabytes when people pass max of usize

pseudo totem
#

Ok... But why to have max size ? I see there's no harm in making it as a optional param.

dreamy zodiac
young plaza
#

well it has optionals, and they can be params :p

dreamy zodiac
#

well yeah, but at that point it makes more sense to just tell people to std.math.maxInt(usize)

pseudo totem
dreamy zodiac
#

it also makes them think about "maybe i should put an upper limit"

pseudo totem
dreamy zodiac
#

if the size of the file is "always" <4096 bytes, then you should definitely cap it to 4096 bytes

pseudo totem
dreamy zodiac
pseudo totem
#

I got that it pre-allocates, consider this method fn readAllAlloc(self: Self, allocator: Allocator, max_size: ?usize,) if max_size is null then array_list will always be created with a capacity of 4096, if it's not null then <4096

dreamy zodiac
#

im failing to see the upside of this API

#

over the current one

pseudo totem
#

As a user I don't care about it just use defaults

dreamy zodiac
#

the current one makes you think about whether you should cap it, and when you do just say "fuck it" and use maxInt, thats very clearly shown in the caller

#

there is no default

#

you should think about this as a user

pseudo totem
#

4096

dreamy zodiac
#

4096 as a max size as the default sounds bad, the default max size should be maxInt if it had to exist, but it doesnt have to exist, and shouldnt exist as a default, the user should have to think about the problem and write code that wont allocate 100gb of ram if you happen to point it to the wrong file

#

like in my game's case, my config files are capped at 16k bytes, since they will realistically never get that big
map files are capped to 1mb
etc etc

#

you should provide a maximum that makes sense

#

if you are reading small text files, you should set a max size that makes sense for that

#

if you are are reading arbitrary big files, you should explicitly set the max size to maxInt

#

if you really dont care, there is always the bail-out of maxInt, but that bailout will be plainly obvious

#

as it should be

pseudo totem
#

From user pov do you think max_size is nothing but the maximum file size ?
Because I donno the file size, I'll think it can go upto size of maxint Is that the assumption ?

dreamy zodiac
#

max_size is the maximum size that the function will read

#

anything bigger than that it wont read

pseudo totem
#

Oh! I get it Thanks

dreamy zodiac
#

np