Is std.Io.Writer.fromArrayList a trap?

[slender pilot]

It crashes on any write, and never resize the ArrayList. It seems like it is exactly the same as std.Io.Writer.fixed.
So 2 questions, how do I get Writer (with allocator) that does not crash on every write?
And what is the difference between fromArrayList(&al), and fixed(al.items)?

[lone walrus]

std.Io.Writer.Allocating is what you're looking for

and to answer the second question, the writer takes ownership of the data in the ArrayList when you call fromArrayList. you can't use that array list afterwards unless you call toArrayList

and fixed(al.items) means, it just uses the array list data as the buffer. the data in the array list will be overwritten and the buffer will not grow

[slender pilot]

oh it does not have type Writer, but Allocating, that is why I did not see it

[lone walrus]

well, yeah it's a different type

[slender pilot]

I still did not get used to that an "Interface" may never be mentioned, but still works...

var compressed_writer = std.Io.Writer.Allocating.init(self.a);
var x = std.compress.flate.Compress.init(&compressed_writer.writer, tmp, .zlib, .default)

🤔 should this code work? std.Io.Writer.Allocating.init does not create big enough buffer for std.compress.flate.Compress.init.

[lone walrus]

initialize it with initCapacity maybe?

the writer i mean

[slender pilot]

But I have no clue, how much I will need, why std.Io.Writer.Allocating does not allocate more, if it have an allocator? Why I need to give it both allocator, and correct guess of size needed?

[lone walrus]

the buffer should grow. that's the point of Allocating

[slender pilot]

I reported it as a bug, because I am 100% sure, that is not my stupidity: https://codeberg.org/ziglang/zig/issues/35757

[lone walrus]

THAT'S NOT A BUG

i think

[slender pilot]

Convince me, and I will close it.

[lucid violet]

How is the compress code supposed to know it's dealing with an allocating writer?

It only has the Writer interface, not the AllocatingWriter

[lone walrus]

what's the value of compressed_writer.writer.buffer.len?

[slender pilot]

It can do try output.ensureUnusedCapacity(9); that is on the Writer "interface".
But how does it know later, to ask it for more space, instead of crashing, if only 9 bytes is provided? Later it somehow knows, but not in the init function

[slender pilot]

0

[lone walrus]

even after initializing it with initCapacity?

[slender pilot]

And I do not think it should be anything else, it should allocate as necessary

[slender pilot]

no, in that case it is whatever the capacity I give it, but why I should guess what initial capacity is required?
it will fill over 70 kB, no matter what capacity I give it, as long as it is bigger than 8.

[lone walrus]

well sostd.compress.flate.Compress.init assumes it's a writer with a fixed buffer

looks like you can't use an Writer.Allocating with it

[slender pilot]

What if I will call initCapacity(9) , and for whatever reason in next version it will break because the condition will change to > 9 for whatever reason?
I think that assert is wrong, it should ask writer if it have enough space, not assert on it's internal state.

[ornate radish]

yeah, the Writer interface is missing something here

[slender pilot]

well you can, if the buffer is already bigger than 8, and it does not matter, if it is 9 B, 10 KB, or 1 MB, all of these work the same, and I still do not think you can convince me, that 0..=8 not working is not a bug

[slender pilot]

🤔 what?

[ornate radish]

it focuses on implementation with fixed size buffer, it doesn't even have a function for ensureCapacity

it only has ensureUnusedCapacity

[slender pilot]

🤔 ok I am missing the detail that would make the second one wrong.

Why would you ever cared about used capacity?

[ornate radish]

if the Writer interface has ensureCapacity, then Compress.init can try to call ensureCapacity and assert on its result

[ornate radish]

well, I assume Compression takes ownership of the buffer of the writer after you pass it in and gives the ownership back after finish is called

so it only cares about the total capacity, not really about the unused capacity

since you can have the total capacity being 100 and the unused being 0

[slender pilot]

🤔 so if I would be writing to fixed buffer of size 1 GB, but would have already used all of it, then UnusedCapacity would be 0, but Capacity would be 1 GB.

[ornate radish]

maybe, this is only an assumption, I haven't checked the implementation of Compression that closely

[slender pilot]

🤔 seems like we have different understanding of what unused is for, I would not want it to overwrite any previously written stuff, imagine it would be writing to a file, and change previously written bytes

[ornate radish]

flush first before you pass it to the Compress writer?

[slender pilot]

🤔 that would seem like weird design, if it would really want to overwrite previous bytes, does any other interface in zig have such crazy footgun? (of overwriting, instead of appending)

[ornate radish]

this line in https://ziglang.org/documentation/master/std/#src/std/compress/flate/Compress.zig makes me think it doesn't try to keep the existing content in the buffer

[slender pilot]

Well that might be another bug, that I am not going to investigate right now.

[ornate radish]

not really sure though

[slender pilot]

me neither, but I am pretty sure, that what I reported is a bug.

[ornate radish]

could be, at least we'll hear from the devs directly

[slender pilot]

I do not believe, that it could be intended, that internal buffer of Writer needs to be prealocated to 9 bytes (or more)

[lone walrus]

did the buffer length increase past the initial capacity?

[slender pilot]

yes

[lone walrus]

i guess you're right. let's see what they say

[fossil fulcrum]

I seem to recall seeing that the compressor stuff have a fixed required minimum buffer size, at one point asserted but maybe broke moving to io.

[slender pilot]

🤔 I do not think assert is correct, if the Writer can allocate more space as needed

[fossil fulcrum]

Just saw the bug in the tracker channel, and I agree.

[gritty gulch]

Seems to me like the buffer starts empty and grows transparently during writes via rebase in the vtable but since the assertion is being done at construction time by the consumer it doesn't get a chance to grow and fails assert

I also do think it is a legacy oversight for the compressed writer case as those asserts would make sense pre io where we all used fixed buffers and allocators as @fossil fulcrum mentioned but now with dynamic growth stuff I feel thats very limiting >~<

[zealous dove]

yes, its not an issue with the writer interface, rather the code that assumes the buffer is fixed in size.

[slender pilot]

https://codeberg.org/ziglang/zig/issues/35757#issuecomment-17428076 @drifting snow seems to disagree

[fossil wyvern]

It always confuses the shit out of me when I see something like std.Io.Writer.Allocating contains an std.Io.Writer

I get it's a code organization thing, and Allocating is basically just "wrapping" Writer and giving some more functionality, but it still throws me every time I see code structured that way. I think my expectation would be to see something like std.Io.AllocatingWriter

[drifting snow]

i do my library this way

[crystal shell]

I think she's right, because implementation should not depend on kind of Writer That's on you to ensure initial capacity ofWriter as Compress.init works with general Writer and not all writer buffers can dynamically grow.
Also Writer doesn't have ensure*Capacity methods, but Writer.Allocating does, so Compress.init cannot call it during construction.

[slender pilot]

but it have ensureUnusedCapacity, so all writes have that.

[crystal shell]

yeah they have it, but defaultRebase does not expand buffer itself. So if you have buffer of 4 bytes it won't grow anyway

[crystal shell]

How can we assume otherwise. iirc only Writer.Allocating can have dynamic buffer size?

[zealous dove]

Writer.Allocating is just an implementation, it is not special.

dont assume there are no other possible use cases, even if writing to a growing buffer is the only usecase
Writer.Allocating is not the only possible way to do that, what if you want a different growth pattern
or different allocation strategy that changing the allocator alone cant provide.
what if your growing buffer is within a more complex allocation

[slender pilot]

😮 wait, it is impossible to write a new writer type?

[zealous dove]

you absolutely can write your own writer implementation

[crystal shell]

I meant in std. Not in general. Ofc you could write a custom allocator writer interface with a growable buffer

[crystal shell]

we will have to implement growable buffers for every Writer interface. Otherwise api depending on it cannot assume they are growable, no? Like fixed buffers can't be growable, but growable ones could be used like fixed

[zealous dove]

what?
APIs dont need to, and should not, assume a writer is growable (if possible).
I am just saying they shouldn't assume they are fixed in size either, you can write code that is compatible with both by just changing asserts on the buffer len to a ensureUnusedCapacity call.

that is the only issue I am aware of

[slender pilot]

🤔 no one is asking for that, the only thing I asked for, was:
for asking the writer, if it have at least 9 bytes of space (and the writer would handle that accordingly, if fixed size, and smaller, just say NO)
instead of asserting on internal implementation of the writer, that it does not know

[crystal shell]

wait i think i get what you're talking about. So just call ensureUnusedCapacity and if it grows - ok, if not return error?

[zealous dove]

it panics if it doesnt grow, which is equivalent behaviour with fixed buffer writers as a direct assert on its length.
but if it can grow, it will, or should, if it doesnt for some reason that is up to the implementation

[crystal shell]

yeah, now i get it

[zealous dove]

I should say, non growing writers, instead of fixed buffer as that is its own thing

[slender pilot]

You can use std.Io.Writer.writableSliceGreedy(9) and assert on it not returning null, if you want to keep the assert in place