Write json array to file | Discord Gophers | Page 1

atomic fulcrum Dec 18, 2022, 9:06 PM

#

i have the entire pinging part

marble mauve Dec 18, 2022, 9:07 PM

#

How are you storing this in the code?

#

If it's just a slice of structs, just pass the whole slice to json.Marshal

#

If it's not a slice of structs, make it into a slice of structs :)

atomic fulcrum Dec 18, 2022, 9:09 PM

#

this is my code: ```go
type Response struct {
IP string json:"ip"
Timestamp string json:"timestamp"
Status string json:"status"
Last string json:"lastping"
}

func main() {
start := time.Now().UnixMilli()

fmt.Println("\n [ Start ] ")

content, err := ioutil.ReadFile("./servers.json")
if err != nil {
    log.Fatal("Error when opening file: ", err)
}

var servers []struct {
    IP string `json:"ip"`
    Port int `json:"port"`
}
err = json.Unmarshal(content, &servers)
if err != nil {
    log.Fatal("Error during Unmarshal(): ", err)
}


var pingout // idk what to put here
for i, data := range servers {
    res, err := PerformPing(data.IP, data.Port)
    if err != nil { fmt.Printf("[%d] (Failed) %s\n", i, err) }
    jsonString, _ := json.Marshal(res)
    pingout = append(pingout, jsonString)
}

final, _ := json.Marshal(pingout)
ioutil.WriteFile("ping_out.json", final, os.ModePerm)

fmt.Println("\n [ End ] ")
fmt.Printf("Took: %dms\n", time.Now().UnixMilli() - start)

}

marble mauve Dec 18, 2022, 9:10 PM

#

var pingout []Response

atomic fulcrum Dec 18, 2022, 9:10 PM

#

oh fromi PerformPing i get the Response struct back

#

Cannot use 'jsonString' (type []byte) as the type Response

marble mauve Dec 18, 2022, 9:11 PM

#

Append the thing before marshaling

#

marshal in the end

atomic fulcrum Dec 18, 2022, 9:11 PM

#

oh ok

#

makes sense i guess

marble mauve Dec 18, 2022, 9:11 PM

#

var pingout []Response
for {
  pingout = append(pingout, res)
}
final, _ := json.Marshal(pingout)

atomic fulcrum Dec 18, 2022, 9:12 PM

#

i get a pointer from PerformPing

#

panic: runtime error: invalid memory address or nil pointer dereference [signal 0xc0000005 code=0x0 addr=0x0 pc=0x46e8ed]

#

nvm im stupid

#

the ping failed so ofc im getting nil

#

but im not skiping the rest of the loop

marble mauve Dec 18, 2022, 9:13 PM

#

continue is your friend

atomic fulcrum Dec 18, 2022, 9:14 PM

#

i just forgot the continue after it fails
if err != nil { fmt.Printf("[%d -> %s] (Failed) %s\n", i, data.IP, err); continue }

#

can i somehow make json.Marshal format the json data it gives me?

marble mauve Dec 18, 2022, 9:14 PM

#

Like indent?

#

MarshalIndent is a thing i think

atomic fulcrum Dec 18, 2022, 9:16 PM

#

marble mauve Like indent?

like this

[
  {
    "some": "thing", 
    "another": "thing"
  }
]

instead of this

[{"some":"thing","another":"thing"}]

marble mauve Dec 18, 2022, 9:16 PM

#

MarshalIndent

atomic fulcrum Dec 18, 2022, 9:16 PM

#

oh ok

#

i have another question

#

the pinging takes a bit of time

#

can i somehow run multiple pings at the same time?

#

because like this, 11 pings take 30secs

#

and thats quite a long time

marble mauve Dec 18, 2022, 9:17 PM

#

Yeah using goroutines and a waitgroup
https://go.dev/tour/concurrency/1

#

I recommend doing the go tour about this, lots of good examples

atomic fulcrum Dec 18, 2022, 9:18 PM

#

sounds interesting

marble mauve Dec 18, 2022, 9:18 PM

#

also this: https://gobyexample.com/waitgroups

atomic fulcrum Dec 18, 2022, 9:18 PM

#

thank you for your help

#

i got a small problem

#

(i think)

#

nvm

#

forgot to add group.Wait()

#

i may have a problem

#

panic: runtime error: slice bounds out of range [-3:]

marble mauve Dec 18, 2022, 9:28 PM

#

That's a panic, runtime error. yep, definitely looks like an error.

#

I bet some code caused this

atomic fulcrum Dec 18, 2022, 9:29 PM

#

it seems to be some library im using

#

goroutine 16 [running]:
github.com/dreamscached/minequery/v2.(*Pinger).ping17ReadStatusResponsePacketPayload(0xc0000564a0?, {0x22c6c08c278, 0xc007740268})
        C:/Users/Sandbox/go/pkg/mod/github.com/dreamscached/minequery/[email protected]/ping_17.go:1841 +0x525
github.com/dreamscached/minequery/v2.(*Pinger).Ping17(0xc0000564a0, {0xc000361d72, 0xd}, 0x0?)
        C:/Users/Sandbox/go/pkg/mod/github.com/dreamscached/minequery/[email protected]/ping_17.go:1757 +0x27b
github.com/dreamscached/minequery/v2.Ping17(...)
        C:/Users/Sandbox/go/pkg/mod/github.com/dreamscached/minequery/[email protected]/ping_17.go:1731
main.main.func1({{0xc000361d72, 0xd}, {0xc000361d80, 0xa}, {0xc00036c100, 0x1, 0x4}}, 0x0?)
        F:/programming/2022/go/server_pinger/main.go:53 +0xe6
created by main.main
        F:/programming/2022/go/server_pinger/main.go:50 +0x608
exit status 2

marble mauve Dec 18, 2022, 9:30 PM

#

Look at the line numbers to know what line caused

atomic fulcrum Dec 18, 2022, 9:30 PM

#

in my main.go file or in the library files?

marble mauve Dec 18, 2022, 9:30 PM

#

Well first in your file

atomic fulcrum Dec 18, 2022, 9:30 PM

#

because the lines in my main.go file is in my PerformPing function

#

pingres, err := minequery.Ping17(data.IP, data.Port)

marble mauve Dec 18, 2022, 9:31 PM

#

Looks like it's in the library, maybe it's not meant to handle concurency

#

Maybe you need a sperate pinger for each site, idk

#

Can't remote-debug it

atomic fulcrum Dec 18, 2022, 9:35 PM

#

the library doesnt use panic and my code doesnt eighter

marble mauve Dec 18, 2022, 9:35 PM

#

It looks like something is trying to access a slice index that doesn't exist

#

Which causes a panic

marble mauve Dec 18, 2022, 9:35 PM

#

atomic fulcrum `panic: runtime error: slice bounds out of range [-3:]`

this

atomic fulcrum Dec 18, 2022, 9:36 PM

#

the last line the error gives me is this: ```
pb.Write(lb[ln-lr.Len() : ln])

marble mauve Dec 18, 2022, 9:36 PM

#

¯_(ツ)_/¯

#

len-lr.Len() seems to come out as negative

#

So panic

atomic fulcrum Dec 18, 2022, 9:37 PM

#

https://github.com/dreamscached/MineQuery/blob/v2/ping_17.go#L1841

GitHub

MineQuery/ping_17.go at v2 · dreamscached/MineQuery

📡 Minecraft Server List Ping library written in Go - MineQuery/ping_17.go at v2 · dreamscached/MineQuery

#

thats the line in the library

marble mauve Dec 18, 2022, 9:38 PM

#

You can open an issue there

atomic fulcrum Dec 18, 2022, 9:38 PM

#

can i somehow "skip" the panic with something like a try {} except () {}?

marble mauve Dec 18, 2022, 9:38 PM

#

There's recover not best practice

#

If this didn't happen when you were doing the requests 1 by 1, maybe the problem is with concurrency, did you try making a separate pinger for each site?

atomic fulcrum Dec 18, 2022, 9:39 PM

#

marble mauve If this didn't happen when you were doing the requests 1 by 1, maybe the proble...

ye but i stopped after 10

#

if i run with the "threads" (idk what to call it) it works on smaller scale like 10 requests

#

but when i go trough my entire list it fails

marble mauve Dec 18, 2022, 9:40 PM

#

fails?

atomic fulcrum Dec 18, 2022, 9:41 PM

#

with the panic from before

marble mauve Dec 18, 2022, 9:41 PM

#

I dunno, can open issue at the library

#

Or try researching recover and hack something together

atomic fulcrum Dec 18, 2022, 9:42 PM

#

can i count how many times i recover from a panic?

violet owl Dec 18, 2022, 9:44 PM

#

You could do that (although it would be a bit hacky) by wrapping the function that panics in an anonymous function that defers a function that recovers and then, if the recover does not return nil, increments a counter using a closure

#

nevermind

#

its just a bad idea

atomic fulcrum Dec 18, 2022, 9:49 PM

#

i think using coroutines wasnt the best idea

violet owl Dec 18, 2022, 9:49 PM

#

you could most likely implement this, but you shuld find another solution if possible

atomic fulcrum Dec 18, 2022, 9:50 PM

#

because i get a lot of i/o timeouts, Couldnt establish connections and could not read response packets

violet owl Dec 18, 2022, 9:51 PM

#

Couldnt you just have a simple wrapper for what you are doing that converts panics to regular errors and then just write the code "normally"

atomic fulcrum Dec 18, 2022, 9:51 PM

#

uh can i print something and then remove that print in the next print call, like print("\rsomething", end="\r") in python?

violet owl Dec 18, 2022, 9:52 PM

#

https://stackoverflow.com/questions/15442292/how-to-have-an-in-place-string-that-updates-on-stdout

Stack Overflow

How to have an in-place string that updates on stdout

I want to output to stdout and have the output "overwrite" the previous output.

For example; if I output On 1/10, I want the next output On 2/10 to overwrite On 1/10. How can I do this?

atomic fulcrum Dec 18, 2022, 9:53 PM

#

violet owl Couldnt you just have a simple wrapper for what you are doing that converts pani...

im using a wrapper to ping servers
and i have my own function that just pings a server removes unwanted attributes from the returned data (or moves the wanted data into another struct) and returns that
then i put that into a slice and output that as json

violet owl Dec 18, 2022, 9:53 PM

#

so you pass the list of addressees to the library and then it panics?

atomic fulcrum Dec 18, 2022, 9:53 PM

#

basically

#

but the library has no panics

violet owl Dec 18, 2022, 9:54 PM

#

what

#

What is the output you are getting

atomic fulcrum Dec 18, 2022, 9:54 PM

#

#

the library has no panics in its code

#

and my code uses no panics

violet owl Dec 18, 2022, 9:54 PM

#

thats not how that works

#

there is intrinsic panic

#

the library also may have imports

#

that can have panics

atomic fulcrum Dec 18, 2022, 9:55 PM

#

the line that panics is this one: https://github.com/dreamscached/MineQuery/blob/v2/ping_17.go#L1841

GitHub

MineQuery/ping_17.go at v2 · dreamscached/MineQuery

📡 Minecraft Server List Ping library written in Go - MineQuery/ping_17.go at v2 · dreamscached/MineQuery

violet owl Dec 18, 2022, 9:55 PM

#

can you give a full stacktrace

atomic fulcrum Dec 18, 2022, 9:55 PM

#

// Read entire packet to a buffer
pb := bytes.NewBuffer(make([]byte, 0, pl))
pb.Write(lb[ln-lr.Len() : ln]) // < panic here
if _, err = io.CopyN(pb, reader, int64(pl)-int64(lr.Len())); err != nil {
    return nil, err

atomic fulcrum Dec 18, 2022, 9:55 PM

#

violet owl can you give a full stacktrace

one sec

#

panic: runtime error: slice bounds out of range [-1:]

goroutine 454 [running]:
github.com/dreamscached/minequery/v2.(*Pinger).ping17ReadStatusResponsePacketPayload(0xc0000564a0?, {0x2785faece38, 0xc0076cc518})
        C:/Users/Sandbox/go/pkg/mod/github.com/dreamscached/minequery/[email protected]/ping_17.go:1841 +0x525
github.com/dreamscached/minequery/v2.(*Pinger).Ping17(0xc0000564a0, {0xc00045e1e0, 0xf}, 0x0?)
        C:/Users/Sandbox/go/pkg/mod/github.com/dreamscached/minequery/[email protected]/ping_17.go:1757 +0x27b
github.com/dreamscached/minequery/v2.Ping17(...)
        C:/Users/Sandbox/go/pkg/mod/github.com/dreamscached/minequery/[email protected]/ping_17.go:1731
main.main.func1({{0xc00045e1e0, 0xf}, {0xc00045e1f0, 0xa}, {0xc00045d100, 0x1, 0x4}}, 0x0?)
        F:/programming/2022/go/server_pinger/main.go:53 +0xe6
created by main.main
        F:/programming/2022/go/server_pinger/main.go:50 +0x608
exit status 2

violet owl Dec 18, 2022, 9:57 PM

#

i am pretty confident this is simply a bug in the library

#

this looks like a lack of fault tolerance

#

Apprently in this case 'ln' is zero while 'lr' is 1, which seems like a case the author simlpy missed

#

leading to this weird negative index slicing operation

#

This is extremely weird though, since the buffer at the beginning 'lb' is a constant size 5 buffer, therefore the implementation of the concrete io.Reader used here must be wrong. Since a read from an io.Reader that did not return an error should return the correct number of elements read, the number of elements read is clearly zero here though, which should not be possible

violet owl Dec 18, 2022, 10:04 PM

#

atomic fulcrum one sec

Are you familiar with dlv by any chance?

atomic fulcrum Dec 18, 2022, 10:04 PM

#

violet owl Are you familiar with dlv by any chance?

no, im new to go

violet owl Dec 18, 2022, 10:06 PM

#

Can you explain how you are using goroutines?

atomic fulcrum Dec 18, 2022, 10:06 PM

#

violet owl Can you explain how you are using goroutines?

can you give me an example of what goroutines are (i suck at remembering names of things)

violet owl Dec 18, 2022, 10:07 PM

#

they allow you to aynchronously run some functions using the go keyword

atomic fulcrum Dec 18, 2022, 10:08 PM

#

oh so go func right?

violet owl Dec 18, 2022, 10:08 PM

#

can you maybe post your current source?

violet owl Dec 18, 2022, 10:08 PM

#

atomic fulcrum oh so `go func` right?

yeah that would be an anonymous function running in a goroutine

atomic fulcrum Dec 18, 2022, 10:08 PM

#

if yes, ```go
go func(data Data, i int) {
// my getting daat code
}(data, i)

#

this: ```go
go func(data Data, i int) {
defer wg.Done()

res, err := PerformPing(data.IP, data.Ports[0].Port)
if err != nil { fmt.Printf("[%d -> %s] (Failed) %s\n", i, data.IP, err); return }
pingout = append(pingout, *res)
}(data, i)

violet owl Dec 18, 2022, 10:10 PM

#

It would be easier if you posted your source

atomic fulcrum Dec 18, 2022, 10:17 PM

#

violet owl It would be easier if you posted your source

k: ```go
package main

type Data struct {
IP string json:"ip"
Timestamp string json:"timestamp"
Ports []struct {
Port int json:"port"
Proto string json:"proto"
Status string json:"status"
Reason string json:"reason"
TTL int json:"ttl"
} json:"ports"
}

type Result struct {
IP string
VersionName string
ProtocolVersion int
OnlinePlayers int
MaxPlayers int
Description Chat17 // type Chat17 interface{} from minequery
Icon image.Image
}

func PerformPing(IP string, Port int) Result {
pingres := minequery.Ping17(IP, Port)

res := Reuslt {
    IP,
    pingres.VersionName,
    pingres.ProtocolVersion,
    pingres.OnlinePlayers,
    pingres.MaxPlayers,
    pingres.Description,
    pingres.Icon,
}

return res

}

func main() {
start := time.Now().UnixMilli()

fmt.Println("\n [ Start ] ")

content, err := ioutil.ReadFile("./server_dump.json")
if err != nil {
    log.Fatal("Error when opening file: ", err)
}

var payload []Data
err = json.Unmarshal(content, &payload)
if err != nil {
    log.Fatal("Error during Unmarshal(): ", err)
}


var wg sync.WaitGroup
var pingout []Result
for i, data := range payload {
    wg.Add(1)

    go func(data Data, i int) {
        defer wg.Done()

        res, err := PerformPing(data.IP, data.Ports[0].Port)
        if err != nil { fmt.Printf("[%d -> %s] (Failed) %s\n", i, data.IP, err); return }
        pingout = append(pingout, *res)
    }(data, i)
}
wg.Wait()

final, _ := json.MarshalIndent(pingout, "", "  ")
err = ioutil.WriteFile("ping_out.json", final, os.ModePerm)
if err != nil { fmt.Printf("(Error -> ioutil.WriteFile) %s\n", err) }

fmt.Println("\n [ End ] ")
fmt.Printf("Took: %dms\n", time.Now().UnixMilli() - start)

}

#

(removed the imports because the messave would've been too long)

#

(also the code is really bad)

violet owl Dec 18, 2022, 10:20 PM

#

alright, could you start by just removing the goroutines

atomic fulcrum Dec 18, 2022, 10:20 PM

#

violet owl alright, could you start by just removing the goroutines

it takes 30secs for 5pings if i remove them (thats why i had them)

#

but without them there is no panic

violet owl Dec 18, 2022, 10:21 PM

#

alright then the library is not thread safe

#

i assume that io.Reader they were using has a threading bug

atomic fulcrum Dec 18, 2022, 10:22 PM

#

violet owl alright then the library is not thread safe

i got some (probably really really bad) thing from your first suggestion, that kinda works

#

var panics int

func wrapAndCountPanics(f func()) {
    defer func() {
        if r := recover(); r != nil {
            panics++
        }
    }()
    f()
}

// ...

// for
go wrapAndCountPanics(func() {
    defer wg.Done()

    res, err := minequery.Ping17(data.IP, data.Ports[0].Port)
    if err != nil { fmt.Printf("[%d -> %s] (Failed) %s\n", i, data.IP, err); return }
    pingout = append(pingout, *res)
})

if panics > 5 {
    panic("More than 5 panics\nexiting") // never gets called
}

violet owl Dec 18, 2022, 10:24 PM

#

i would strongly recommend not using this

atomic fulcrum Dec 18, 2022, 10:24 PM

#

there are never more than 5 panics
i think its probably just the one

violet owl Dec 18, 2022, 10:24 PM

#

you should look for a different library that is thread safe

atomic fulcrum Dec 18, 2022, 10:25 PM

#

if i print the panics variable at the end, it gives me 1

#

so i think its just some weird server response

violet owl Dec 18, 2022, 10:25 PM

#

i dont think so, this will most likely give you wrong outputs

atomic fulcrum Dec 18, 2022, 10:26 PM

#

why?

violet owl Dec 18, 2022, 10:27 PM

#

the reader panics are most likely not because of a response but because of threading issues

atomic fulcrum Dec 18, 2022, 10:29 PM

#

but if it only happens once i can just ignore it, right?

#

because im checking ca 2.5k servers

violet owl Dec 18, 2022, 10:29 PM

#

it might happen in the 'wrong' thread

#

so the result might be wrong

atomic fulcrum Dec 18, 2022, 10:30 PM

#

violet owl it might happen in the 'wrong' thread

?

#

h.... a...

violet owl Dec 18, 2022, 10:30 PM

#

i am sorry but i dont have the time to explain multithreading right now

atomic fulcrum Dec 18, 2022, 10:31 PM

#

ok

#Write json array to file