How does a reader decide on size of its byte slice? (trying to inject text after EOF) | Discord Gophers | Page 1

pine mesa May 12, 2023, 6:13 PM

#

What I'm trying to do is a bit complex and specific maybe but any general info could help.

I want to inject javascript at the end of html files. I'm wrapping the reader used in a http fileserver. Code works but only for some pages seemingly at random. Sometimes it hits EOF and the byte slice is the exact length for appending my javascript, but sometimes len(b)=1 which makes no sense to me.

My reader:

func (i injectedFile) Read(b []byte) (n int, err error) {
    fi, _ := i.Stat()

    n, err = i.file.Read(b)

    if err == io.EOF && path.Ext(fi.Name()) == ".html" {
        for i, v := range JS {
            b[i] = v
        }
        n += len(JS)
    }

    return n, err
}

I've wrapped the file info size to account for increased size:

func (ifi injectedFileInfo) Size() int64 {
    if path.Ext(ifi.Name()) == ".html" {
        return ifi.fileInfo.Size() + int64(len(JS))
    } else {
        return ifi.fileInfo.Size()
    }
}

#

A successfull read data:

file size (with javascript) 1015
javascript size 251
reader byte slice size 512
bytes read 512
error <nil>

file size (with javascript) 1015
javascript size 251
reader byte slice size 503
bytes read 252
error <nil>

file size (with javascript) 1015
javascript size 251
reader byte slice size 251
bytes read 0
error EOF
-->HERE I INJECT JAVASCRIPT AND HAVE PERFECT SIZE TO DO SO

An unsuccessfull one:

file size (with javascript) 762
javascript size 251
reader byte slice size 512
bytes read 511
error <nil>

file size (with javascript) 762
javascript size 251
reader byte slice size 1
bytes read 0
error EOF
-->ERROR HERE BECAUSE b DOES NOT FIT JS

#

So in the first example we see read is called three times and works as expected, but in the second it was called twice but it left out one byte reading the file and didn't provide enough space for the JS.

rustic hearth May 12, 2023, 6:17 PM

#

How does a reader decide on size of its byte slice?
it doesn't, it reads data into the slice it was given, the caller decides what the (maximum) size is

#

if I give you a slice with len() = 50, you can read up to 50 bytes

pine mesa May 12, 2023, 6:21 PM

#

rustic hearth > How does a reader decide on size of its byte slice? it doesn't, it reads data ...

Yeah that makes a lot of sense. A bit stupid of me giving the reader so much power. I guess in this case since I'm using http.FileSystem and http.FileServer a lot of this is abstracted away from me so it's a bit of a mystery what's going on. On some pages it leaves a byte for the second calling and hits EOF even if it doesn't end up reading this byte... hmmmm

hollow tree May 12, 2023, 6:24 PM

#

you can inject the response with middleware

#

func incertMiddleware(next http.Handler) http.Handler {
  return http.HandlerFunc(func(w, r) {
    next.ServeHTTP(w,r)
    if path.Ext(r.URL.Path()) == ".html" {
      w.Write([]byte(JS))
    }
  })
}

fs:=http.FileServer(http.Dir("dirname"))
mux.Handle("/", incertMiddleware(fs))

#

something like that

pine mesa May 12, 2023, 6:29 PM

#

OK cool I'll take a look at that

#

I've spent so much time working on this reader that it would suck to throw it away though but yeah it is what it is, cool if there is a simpler solution 😆

rustic hearth May 12, 2023, 6:30 PM

#

what are you actually trying to do?

pine mesa May 12, 2023, 6:33 PM

#

rustic hearth what are you actually trying to do?

A normal static local webserver on a directory. But then it injects the html pages with some custom javascript that can refresh pages using websockets that are triggered by file changes using a file watcher. It's for convenient development of static websites so when you update the css for example the browser refreshes automatically to reflect that.

rustic hearth May 12, 2023, 6:36 PM

#

seems like performance/efficiency is not a huge deal, is that correct?

pine mesa May 12, 2023, 6:37 PM

#

rustic hearth seems like performance/efficiency is not a huge deal, is that correct?

Yeah but it depends, but yeah it's only going to be a local thing, but I do want it to be fast/correct ofc but not overly picky...

#

I suspect I'm really close to the solution (since it works perfectly for some pages) but could be wrong

rustic hearth May 12, 2023, 6:43 PM

#

then I think the most sensible solution is to implement http.FileSystem.
In your Open method you can read the file into memory, do the replacement, and then serve that. Something along the lines of (writing it on the fly, untested): ```go
type replaceFs struct{}

func (replaceFs) Open(name string) (http.File, error) {
return newReplaceFile(name)
}

type replaceFile struct {
contents *bytes.Reader
*os.File // implements most of the http.File interface, but the file size will be wrong you need to deal with that
}

func (r *replaceFile) Read(d []byte) (int, error) { return r.contents.Read(d) }

func newReplaceFile(name string) (*replaceFile, error) {
f, err := os.Open(name)
if err != nil {
return nil, err
}

data, err := io.ReadAll(f)
if err != nil {
    return nil, err
}

replaced := bytes.Replace(data, needle, replacement, 1)

return &replaceFile{
    File:   f,
    contents: bytes.NewReader(replaced),
}, nil

}

#

that should be about right

pine mesa May 12, 2023, 6:44 PM

#

Thank you I will take a look at this

#

In my solution I am using filesystem but I'm basically overwriting methods (using wrapping of structs) all the way down to fs.FileInfo and file http.File:

type injectedFile struct {
    file http.File
}

func (i injectedFile) Seek(offset int64, whence int) (int64, error) {
    return i.file.Seek(offset, whence)
}

func (i injectedFile) Readdir(count int) ([]fs.FileInfo, error) {
    return i.file.Readdir(count)
}

...

type injectedFileInfo struct {
    fileInfo fs.FileInfo
}

func (ifi injectedFileInfo) Name() string {
    return ifi.fileInfo.Name()
}

func (ifi injectedFileInfo) Size() int64 {
    if path.Ext(ifi.Name()) == ".html" {
        return ifi.fileInfo.Size() + int64(len(JS))
    } else {
        return ifi.fileInfo.Size()
    }
}

...

type injectedFileSystem struct {
    fileSystem http.FileSystem
}

func (i *injectedFileSystem) Open(name string) (http.File, error) {
    f, err := i.fileSystem.Open(name) 
    if err != nil {
        return nil, err
    }
    return injectedFile{file: f}, nil
}

#

Leaving out much code but that's an overview

#

But I'll take a look at your suggesting when I have time

rustic hearth May 12, 2023, 6:49 PM

#

Oh I didn't notice the interface required a Readdir and Stat method, you'll need to add those

#

I've updated the snippet, bonus: it almost compiles now 🙃

pine mesa May 15, 2023, 2:57 PM

#

rustic hearth I've updated the snippet, bonus: it almost compiles now 🙃

Thank you for the snippet but I've decided to try my way since I found out I was super close and my way is probably faster. What I've found out is that some pages for some reason is consistently displayed as a blank page and there is an error in the browser: ERR_CONTENT_LENGTH_MISMATCH 200 (OK) which apparently means "a byte difference between the size of the response in the header compared to the actual size of the response document". The funny thing though is that if I right click on the blank page and go to "view page source" the source shows up exactly as intended with the injected script at the bottom! So there might be just some small detail where it skips a byte or something which causes the issue....

My reader hasn't changed much

func (i injectedFile) Read(b []byte) (n int, err error) {
    fi, _ := i.Stat()

    n, err = i.file.Read(b)

    if err == io.EOF && path.Ext(fi.Name()) == ".html" && len(b) == len(JS) {
        for i, v := range JS {
            b[i] = v
        }
        n += len(JS)
    }

    return n, err
}

#

IF ANYONE HAS ANY IDEA WHAT CAUSES THIS LET ME KNOW

hollow tree May 15, 2023, 3:02 PM

#

http file server set the content length with value returned by stat.
i think you need to mock or implement your own http.FS and http.File

pine mesa May 15, 2023, 3:03 PM

#

hollow tree http file server set the content length with value returned by stat. i think you...

I have done this. I use http.File and I have a stat that adds the size of JS for html pages. I overwrite this method.

#

It's possible it has to do with the fact I only insert the JS when I have a byte slice that's the size of the JS, but without this check it crashes because it doesn't hit EOF correctly, because for some reason it sometimes gives a byte slice that's the length of 1 and it doesn't read this byte... and so there is something fishy there

rustic hearth May 15, 2023, 3:05 PM

#

pine mesa Thank you for the snippet but I've decided to try my way since I found out I was...

I've decided to try my way since I found out I was super close and my way is probably faster.
It's also incorrect and "works" entirely by accident 🙃

pine mesa May 15, 2023, 3:06 PM

#

rustic hearth > I've decided to try my way since I found out I was super close and my way is p...

Yours or mine? xd

rustic hearth May 15, 2023, 3:06 PM

#

yours

#

as I explained previously, the size of b is more or less arbitrary, if it ends up being equal to some value is pure luck

pine mesa May 15, 2023, 3:07 PM

#

Yeah but my setup is correct so I just need to change the reader a little then

rustic hearth May 15, 2023, 3:07 PM

#

it's also slower as you are stating per read

rustic hearth May 15, 2023, 3:07 PM

#

pine mesa Yeah but my setup is correct so I just need to change the reader a little then

no it isnt

pine mesa May 15, 2023, 3:08 PM

#

Yeah it's a good point with the stat per read

rustic hearth May 15, 2023, 3:09 PM

#

that's the most irrelevant bit mate, your code is simply incorrect

#

if it were faster it would just be wrong faster 🙃

pine mesa May 15, 2023, 3:10 PM

#

type injectedFileInfo struct {
    fileInfo fs.FileInfo
}

func (ifi injectedFileInfo) Name() string {
    return ifi.fileInfo.Name()
}

func (ifi injectedFileInfo) Size() int64 {
    if path.Ext(ifi.Name()) == ".html" {
        return ifi.fileInfo.Size() + int64(len(JS))
    } else {
        return ifi.fileInfo.Size()
    }
}

func (ifi injectedFileInfo) Mode() fs.FileMode {
    return ifi.fileInfo.Mode()
}

func (ifi injectedFileInfo) ModTime() time.Time {
    return ifi.fileInfo.ModTime()
}

func (ifi injectedFileInfo) IsDir() bool {
    return ifi.fileInfo.IsDir()
}

func (ifi injectedFileInfo) Sys() any {
    return ifi.fileInfo.Sys()
}

type injectedFile struct {
    file http.File
}

func (i injectedFile) Seek(offset int64, whence int) (int64, error) {
    return i.file.Seek(offset, whence)
}

func (i injectedFile) Readdir(count int) ([]fs.FileInfo, error) {
    return i.file.Readdir(count)
}

func (i injectedFile) Stat() (fs.FileInfo, error) {
    oldFi, err := i.file.Stat()
    if err != nil {
        return oldFi, err
    }
    newFi := injectedFileInfo{oldFi}
    return newFi, nil
}

func (i injectedFile) Close() error {
    return i.file.Close()
}

func (i injectedFile) Read(b []byte) (n int, err error) {
    fi, _ := i.Stat()

    n, err = i.file.Read(b)

    if err == io.EOF && path.Ext(fi.Name()) == ".html" && len(b) == len(JS) {
        for i, v := range JS {
            b[i] = v
        }
        n += len(JS)
    }

    return n, err
}

type injectedFileSystem struct {
    fileSystem http.FileSystem
}

func (i *injectedFileSystem) Open(name string) (http.File, error) {
    f, err := i.fileSystem.Open(name)
    if err != nil {
        return nil, err
    }
    return injectedFile{file: f}, nil
}

#

@rustic hearth this is what it looks like

rustic hearth May 15, 2023, 3:12 PM

#

what are you having trouble with?

pine mesa May 15, 2023, 3:13 PM

#

The content length mismatch from the incorrect reader

rustic hearth May 15, 2023, 3:13 PM

#

yes, because your code is wrong for the aforementioned reasons

#

I'm asking what are you having trouble understanding

#

even the premise for the question is dubius, you can't inject javascript after the html ends

pine mesa May 15, 2023, 3:17 PM

#

It makes sense to me that b is arbitrary and that checking the length like I do is incorrect. Read is designed to be called multiple times I get that. I think actually rn that I need to find a way to not add all the JS at once but be able to read it in chunks maybe.

pine mesa May 15, 2023, 3:18 PM

#

rustic hearth even the premise for the question is dubius, you can't inject javascript *after*...

It works to append javascript that's inside script tags to the end of an html file

pine mesa May 15, 2023, 3:19 PM

#

pine mesa It makes sense to me that b is arbitrary and that checking the length like I do ...

This the main problem?

rustic hearth May 15, 2023, 3:20 PM

#

pine mesa It works to append javascript that's inside script tags to the end of an html fi...

I highly doubt that's sound, as that is invalid html. It may be that the recovery scenario prescribed by the spec results in correct behaviour but I'm not gonna be the one to check 🙂

rustic hearth May 15, 2023, 3:21 PM

#

pine mesa This the main problem?

the main problem is you have no idea what you're writing to/over.
b can be any length, and it may have been written to already.

pine mesa May 15, 2023, 3:24 PM

#

rustic hearth the main problem is you have no idea what you're writing to/over. `b` can be any...

Well, that's why I'm checking for EOF so I know that (the old) read has finished reading the html file and then afterwards I know that there are going to be enough b's to cover the size of the JS because of my overwritten stat. Correct?

rustic hearth May 15, 2023, 3:25 PM

#

incorrect

pine mesa May 15, 2023, 3:25 PM

#

gophersadsweat

rustic hearth May 15, 2023, 3:25 PM

#

b can be any length, and it may have been written to already.

pine mesa May 15, 2023, 3:25 PM

#

Written to by what?

rustic hearth May 15, 2023, 3:25 PM

#

by your call to file.read

#

the fact that it returned EOF does not mean it didn't write anything

pine mesa May 15, 2023, 3:26 PM

#

yeah that makes sense, so it's not an empty b necessarily, n could be > 0.

#

do I have to get an index for the first empty spot maybe? hmm

rustic hearth May 15, 2023, 3:37 PM

#

if you insist on producing invalid html you need to detect when the html ended and then "switch" to serving your string

pine mesa May 15, 2023, 4:05 PM

#

I guess I can use EOF in conjunction with n to do that, at least the first time around but after idk, I guess I have to play around a bit. Thank you so much for your help. I've spent a ton of time on this project... I've watched "in depth" videos about readers and done various things with them but apparently didn't fully grasp the basics. Also I'm quite blind since I've spent so much time with this code.

pine mesa May 16, 2023, 3:26 PM

#

type injectedFile struct {
    file http.File
    JS   *bytes.Reader
}

func (i injectedFile) Read(b []byte) (n int, err error) {
    if !doneReadHTML {
        n, err = i.file.Read(b)
        if err == io.EOF {
            doneReadHTML = true
        }
        return n, err
    } else {
        n, err = i.JS.Read(b)
        return n, err
    }
}

My first attempt, doesn't work... doneReadHTML is global

rustic hearth May 16, 2023, 3:28 PM

#

why would it be a global?

pine mesa May 16, 2023, 3:31 PM

#

it needs to be global so that it can be referenced again, I wanted to put it in the file struct but that doesn't work since read cannot have a pointer receiver and so changing the bool wouldn't do anything (just forgotten on next read call)

#

but right now it reads correctly the file but the last chunk of it is read twice for some reason and the JS doesn't show up, hmmm

rustic hearth May 16, 2023, 3:34 PM

#

pine mesa it needs to be global so that it can be referenced again, I wanted to put it in ...

it needs to be global so that it can be referenced again
I don't understand what you mean by this

#

to be clear, not only does it not need to be a global, it mustn't be a global, I want to understand your reasoning

rustic hearth May 16, 2023, 3:36 PM

#

pine mesa but right now it reads correctly the file but the last chunk of it is read twice...

think through what you are doing when you see an EOF

pine mesa May 16, 2023, 3:36 PM

#

It's not global to the program, but to the package. It needs to be outside so that I can "flip the switch" so next time read is called I can get info wheter html was read or not, if it was in local scope it wouldn't stay flipped if u know what I mean

rustic hearth May 16, 2023, 3:36 PM

#

why would it "not stay flipped"?

#

I'll give you a hint, would b here be flipped? go var b bool flip(b)

pine mesa May 16, 2023, 3:38 PM

#

This is what I wanted but u can see that there is an error (ineffective assignment to field injectedFile.doneReadHTML)

#

rustic hearth May 16, 2023, 3:38 PM

#

precisely, see my last msg

pine mesa May 16, 2023, 3:39 PM

#

flip(b) same as b = true I guess, I think it would stay flipped but maybe not since ur asking the question xd

rustic hearth May 16, 2023, 3:40 PM

#

https://go.dev/play/p/oAaFNEE1fNl

#

can you tell me why?

pine mesa May 16, 2023, 3:43 PM

#

package main

import "fmt"

var (
    b = false
)

func main() {
    change()
    fmt.Println(b)
}

func change() {
    b = true
}

👆 This is more what I have and it does print true

#

as I expected

rustic hearth May 16, 2023, 3:44 PM

#

I am aware that's what you have

#

that code is incorrect in the context of the problem you're trying to solve

rustic hearth May 16, 2023, 3:45 PM

#

rustic hearth can you tell me why?

^

pine mesa May 16, 2023, 3:46 PM

#

rustic hearth ^

I don't see the connection with my code but yeah main() and flip() have different scopes and so b is local to their scope only, u would have to use a pointer in flip in order to flip the actual b

rustic hearth May 16, 2023, 3:47 PM

#

in particular, when calling flip b is copied

#

in func Read(i injectedFile, b []byte) (n int, err error) what's happening to i?

pine mesa May 16, 2023, 3:49 PM

#

It's copied?, but this is why I was talking about the pointer receivers earlier (which is not possible for how readers are supposed to be though so not an option). This is why I put the bool outside...

#

When opening a new html file I make sure to flip it to false again

rustic hearth May 16, 2023, 3:51 PM

#

pine mesa It's copied?, but this is why I was talking about the pointer receivers earlier ...

It's copied
yes it is

talking about the pointer receivers earlier
I don't recall ever seeing this, feel free to link me to the msg

which is not possible for how readers are supposed to be though so not an option
I don't see why this would be true

pine mesa May 16, 2023, 3:52 PM

#

#1106645438217596989 message

rustic hearth May 16, 2023, 3:52 PM

#

oh you edited that message, I did not see the edit

#

then elaborate on why using a pointer receiver is not possible

#

take into account I gave you a mostly functioning (some details missing but working otherwise) solution that used them

pine mesa May 16, 2023, 3:53 PM

#

cannot use injectedFile{…} (value of type injectedFile) as http.File value in return statement: injectedFile does not implement http.File (method Read has pointer receiver) hmmm

#

I can do this it seems like yeah now that u say it

rustic hearth May 16, 2023, 3:54 PM

#

that compiler error is saying you cannot use injectedFile not that *injectedFile cannot implement an interface

#

you can't use injectedFile because injectedFile does not implement that interface, *injectedFile does

pine mesa May 16, 2023, 4:30 PM

#

I think what happens is that it fills the space reserved for the javascript with a repeat of the last chunk of the html, but the code looks correct to me, hmmmmm

#

I FIGURED IT OUT IT SEEMS LIKE YAY

#

func (i *injectedFile) Read(b []byte) (n int, err error) {
    if !i.doneReadHTML {
        n, err = i.file.Read(b)
        if err == io.EOF {
            i.doneReadHTML = true
            n, err = i.JS.Read(b)
            return n, err
        } else {
            return n, err
        }
    } else {
        n, err = i.JS.Read(b)
        return n, err
    }
}

#

I'm just wondering if need to not throw away the first n if EOF is hit, maybe oldn+newn?

rustic hearth May 16, 2023, 4:39 PM

#

when you reach EOF you are overwriting what you just wrote

#

your first attempt was closer to correct, albeit still not quite there

rustic hearth May 16, 2023, 4:40 PM

#

rustic hearth think through what you are doing when you see an EOF

^

pine mesa May 16, 2023, 4:40 PM

#

Ok gophersadsweat . The more incorrect code I make the more it works xd this one works perfectly from limited testing

#

hmm

rustic hearth May 16, 2023, 4:41 PM

#

that's not impossible

pine mesa May 16, 2023, 4:41 PM

#

func (i *injectedFile) Read(b []byte) (n int, err error) {
    if !i.doneReadHTML {
        n, err = i.file.Read(b)
        if err == io.EOF {
            i.doneReadHTML = true
            jn, err := i.JS.Read(b)
            return n + jn, err
        } else {
            return n, err
        }
    } else {
        n, err = i.JS.Read(b)
        return n, err
    }
}

rustic hearth May 16, 2023, 4:41 PM

#

it's still incorrect as an EOF is not required to be accompanied by a 0 n

pine mesa May 16, 2023, 4:42 PM

#

Ik it's incorrect but it's my first instinct

rustic hearth May 16, 2023, 4:42 PM

#

rustic hearth your first attempt was closer to correct, albeit still not quite there

by this I mean, go back to what you wrote then

pine mesa May 16, 2023, 4:42 PM

#

rustic hearth it's still incorrect as an EOF is not required to be accompanied by a 0 n

isn't this what I do here #1106645438217596989 message

rustic hearth May 16, 2023, 4:43 PM

#

I don't understand that question

pine mesa May 16, 2023, 4:43 PM

#

I mean I take into account that n, add it to returned n

#

but I see your point about overwriting b since it's the same b

rustic hearth May 16, 2023, 4:45 PM

#

rustic hearth when you reach EOF you are overwriting what you just wrote

I mean exactly what I wrote, I'm not talking about n, I'm talking about b, again, your previous attempt was objectively closer to correct

rustic hearth May 16, 2023, 4:47 PM

#

pine mesa ```go type injectedFile struct { file http.File JS *bytes.Reader } fu...

what I mean by previous

pine mesa May 16, 2023, 4:47 PM

#

Well I know the issue with my previous code is that it fills the space that supposed to have javascript with a repeat of some html

rustic hearth May 16, 2023, 4:47 PM

#

rustic hearth think through what you are doing when you see an EOF

to which this was my reply

pine mesa May 16, 2023, 4:50 PM

#

It makes sense to me to return n, err after EOF is hit idk what else to do, maybe I need to not return but rather start reading on the JS but not from the beginning of b but from n, is this on the right track u think?

rustic hearth May 16, 2023, 4:50 PM

#

what does EOF mean?

pine mesa May 16, 2023, 4:52 PM

#

end of file, when the html file has been fully read in this case. This was always my assumption and I did print the output and it confirmed that EOF is hit after html is read

rustic hearth May 16, 2023, 4:52 PM

#

pine mesa ```go type injectedFile struct { file http.File JS *bytes.Reader } fu...

so, when should you return EOF here?

pine mesa May 16, 2023, 4:53 PM

#

I SEE

func (i *injectedFile) Read(b []byte) (n int, err error) {
    if !i.doneReadHTML {
        n, err = i.file.Read(b)
        if err == io.EOF {
            i.doneReadHTML = true
        }
        return n, nil
    } else {
        n, err = i.JS.Read(b)
        return n, err
    }
}

#

It works 😆

#

I guess I should account for if err is not nil and not eof though but yeah I get what you meant now

#

Thanks

rustic hearth May 16, 2023, 5:02 PM

#

fwiw your Seek implementation is also wrong, you do not account for seeks that start after the original file contents

hollow tree May 16, 2023, 5:21 PM

#

i'd rather make a wrapper that remove the content-length header so that seek would never be called.

#

though, range request wouldn't work. but you don't need it anyway.

subtle marlin May 16, 2023, 6:57 PM

#

@pine mesa btw I havn't followed the conversation but would this solves what you want to do: reader := io.MultiReader(html, js) ?

pine mesa May 16, 2023, 6:59 PM

#

subtle marlin <@456226577798135808> btw I havn't followed the conversation but would this solv...

I does work now with the code above, but thank you I hadn't heard of multireader might try it

#

hmm I didn't see that about seek

#

hmmm

#How does a reader decide on size of its byte slice? (trying to inject text after EOF)