Understanding file.Read

[twin plaza]

var file, err = os.OpenFile(Big, os.O_RDONLY, 0666)
    
if err != nil {
  log.Fatal(err)
}

defer file.Close()

var info, _ = file.Stat()

var buf = make([]byte, info.Size())

for {
  _, err = file.Read(buf)
  if err == io.EOF {
    break
  }
}

i'm dealing with a rather large file (8GB) that i want to completely read into the buf... am i doing this correctly, or does go impose some kind of read limit (i've encountered this in C# unfortunately) where you can only read a certain amount of data from a file?

[craggy iron]

just use io.ReadAll() if you want to read it all

$[io.ReadAll]

[pearl dagger]

[email protected]: ReadAll

func ReadAll(r Reader) ([]byte, error)```
ReadAll reads from r until an error or EOF and returns the data it read. A successful call returns err == nil, not err == EOF. Because ReadAll is defined to read from src until EOF, it does not treat an EOF from Read as an error to be reported.

[craggy iron]

you wouldn't need a for loop for a regular file but if you had a for loop (like you'd need for a socket) it'd need to advance in your buffer instead of reading to the start of it each time (and ignoring the n being returned by Read())

[twin plaza]

so does my current implementation not read the entire file into buf?

because io.ReadAll takes a LOT longer to read the entire file, but if that means that my implementation is perhaps overwriting data, then i guess i have to change my approach...

[dusky knot]

Im guessing, but readall might be slower because ot needs to keep resizing the internal buffer

[craggy iron]

remove the for and use $[io.ReadFull] so there is no reallocation (which is what makes ReadAll slower)

[pearl dagger]

[email protected]: ReadFull

func ReadFull(r Reader, buf []byte) (n int, err error)```
ReadFull reads exactly len(buf) bytes from r into buf. It returns the number of bytes copied and an error if fewer bytes were read. The error is EOF only if no bytes were read. If an EOF happens after reading some but not all the bytes, ReadFull returns ErrUnexpectedEOF. On return, n == len(buf) if and only if err == nil. If r returns an error having read at least len(buf) bytes, the error is dropped.

[dusky knot]

You can try ReadFull

And oass your own buffer

[craggy iron]

but for a regular file, ReadFull and Read are the same (again, not for a socket etc)

also and maybe more importantly... if you can process your file as a stream instead of loading all 8Gb in memory before doing anything, that'd probably be much better

[twin plaza]

so... my approach works?

i apologize for bugging about this, but ReadFull is still slower for me than using file.Read in the for loop...

i'm trying a similar version of the 1 billion row challenge, so all i do is read the file initially, parse it and then output the parsed data.

[craggy iron]

it doesn't work because of the for loop (read my first comment)

[twin plaza]

oh

[craggy iron]

why did you put os.O_SYNC btw?

[craggy iron]

yeah so you'll never get good performance by reading the whole thing before starting to process

[twin plaza]

as far as i can tell this is for write operations only, so there isn't a reason i'm pretty sure... my bad

[craggy iron]

use $[bufio.NewScanner]

[pearl dagger]

[email protected]: NewScanner

func NewScanner(r io.Reader) *Scanner```
NewScanner returns a new Scanner to read from r. The split function defaults to ScanLines.

[twin plaza]

and then just use for scanner.Scan() and read scanner.Bytes() ?

[craggy iron]

depends on your input... is it line based (text) or...

[twin plaza]

the input is text, and line based, yes

and my current approach relies on the read data being in byte form

@craggy iron do you have any suggestions on how to approach this?

[craggy iron]

yes so do use NewScanner and .Scan

it's made exactly for streaming data with line of unknown/variable length